<div dir="ltr">Please, file a bug about that on <a href="https://bugs.launchpad.net/neutron/+bugs">https://bugs.launchpad.net/neutron/+bugs</a> with tag "l3-ha".</div><br><div class="gmail_quote"><div dir="ltr">On Thu, Nov 30, 2017 at 6:00 AM Ajay Kalambur (akalambu) <<a href="mailto:akalambu@cisco.com">akalambu@cisco.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div>
<div>
<div>I noticed that this happens when the router HA interface shows status: Down what can cause the ha interface to do down</div>
<div><br>
</div>
<div>Ajay</div>
<div><br>
</div>
<div>
<div id="m_-1008888023407271136MAC_OUTLOOK_SIGNATURE"></div>
</div>
</div>
</div>
<div><br>
</div>
<span id="m_-1008888023407271136OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri;font-size:12pt;text-align:left;color:black;BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0in;PADDING-LEFT:0in;PADDING-RIGHT:0in;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt">
<span style="font-weight:bold">From: </span>Ajay Kalambur <<a href="mailto:akalambu@cisco.com" target="_blank">akalambu@cisco.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, November 29, 2017 at 4:14 PM<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>[openstack-dev] [neutron] Neutron L3 agent/Keepalived<br>
</div></span></div><div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif"><span id="m_-1008888023407271136OLK_SRC_BODY_SECTION">
<div><br>
</div>
<span>
<div>
<div style="word-wrap:break-word">
<div>
<div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
Hi</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
I have a case where after running the system for a while I see that floating ip association API gets accepted and no Errors in neutron l3 logs</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
But when I go into the qrouter namespace and check the qg- interface the floating ip is not added</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
Also the keepalived.conf is not updated and SIGUP of the keepalived process is not done</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
So whats the place to look in this case. What is the flow in neutron l3 agent who adds the floating ip to the qg- interface </div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
I see the router update notification received</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
Key log snippet attached. As you can see SIGUP of keepalived is missing and also I confirmed keepalived configs are not updated</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<br>
</div>
<div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:08.883 78 DEBUG oslo_messaging._drivers.amqpdriver [-] received message with unique_id: b3886b998c3049e799170bb5351300d1 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:269</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:08.885 78 DEBUG neutron.agent.l3.agent [req-6c505d32-2d1a-4392-8ea3-0bb888397b9e e759eebecc4648b4964d4ecf439dc0ff 13b4f6fb188048ba9bbf211344e3342f - - -] Got routers updated notification :[u'a1a59e5f-d74e-404d-a3aa-1667c4442aed']
routers_updated /usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:413</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:08.886 78 DEBUG neutron.agent.l3.agent [-] Starting router update for a1a59e5f-d74e-404d-a3aa-1667c4442aed, action None, priority 0 _process_router_update /usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:487</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:08.886 78 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 92eb015c8c84489681b1efbe949fab8b exchange 'neutron' topic 'q-l3-plugin' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:568</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.571 78 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 92eb015c8c84489681b1efbe949fab8b __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:416</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.571 78 DEBUG neutron.callbacks.manager [-] Notify callbacks [] for router, before_update _notify_loop /usr/lib/python2.7/site-packages/neutron/callbacks/manager.py:142</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.572 78 DEBUG neutron.agent.l3.router_info [-] process router updates process /usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:1105</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.572 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'find', '/sys/class/net', '-maxdepth', '1', '-type',
'l', '-printf', '%f '] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.709 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.710 78 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" acquired by "neutron.agent.linux.pd.sync_router" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.710 78 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" released by "neutron.agent.linux.pd.sync_router" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.710 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'find', '/sys/class/net', '-maxdepth', '1', '-type',
'l', '-printf', '%f '] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.845 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.846 78 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "iptables-qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.846 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'iptables-save'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.983 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.986 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'iptables-restore', '-n'] execute_rootwrap_daemon
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.120 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.120 78 DEBUG neutron.agent.linux.iptables_manager [-] IPTablesManager.apply completed with success. 7 iptables commands were issued _apply_synchronized /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:576</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.121 78 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "iptables-qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225</font></div>
<div><font face="Calibri,sans-serif"><b>2017-11-29 14:32:10.121 78 DEBUG neutron.agent.l3.router_info [-] Floating ip 5966c8b3-ff5c-4f01-91a6-f81d215bdeca added, status ACTIVE process_floating_ip_addresses /usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:347</b></font></div>
<div><font face="Calibri,sans-serif"><b>2017-11-29 14:32:10.122 78 DEBUG neutron.agent.l3.router_info [-] Sending floating ip statuses: {u'5966c8b3-ff5c-4f01-91a6-f81d215bdeca': 'ACTIVE'} update_fip_statuses /usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:915</b></font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.122 78 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 18e6ceffec9549a5bd51f65551b65596 exchange 'neutron' topic 'q-l3-plugin' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:568</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.150 78 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 18e6ceffec9549a5bd51f65551b65596 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:416</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.151 78 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "iptables-qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.151 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'iptables-save'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.301 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.304 78 DEBUG neutron.agent.linux.iptables_manager [-] IPTablesManager.apply completed with success. 0 iptables commands were issued _apply_synchronized /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:576</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.304 78 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "iptables-qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.304 78 DEBUG neutron.callbacks.manager [-] Notify callbacks [('neutron.agent.linux.pd.update_router--9223372036851355335', <function update_router at 0x3431398>), ('neutron.agent.metadata.driver.after_router_updated-3681661',
<function after_router_updated at 0x382d7d0>)] for router, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron/callbacks/manager.py:142</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.305 78 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" acquired by "neutron.agent.linux.pd.update_router" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.305 78 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" released by "neutron.agent.linux.pd.update_router" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.305 78 DEBUG neutron.agent.l3.agent [-] Finished a router update for a1a59e5f-d74e-404d-a3aa-1667c4442aed _process_router_update /usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:536</font></div>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<div id="m_-1008888023407271136"></div>
</div>
</div>
</div>
</div>
</div>
</span></span></div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div><div dir="ltr">-- <br></div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Regards,<br>Ann Taraday</div></div>