<div dir="ltr">Which neutron branch you are using? <div>May be this bug <a href="https://bugs.launchpad.net/neutron/+bug/1723848">https://bugs.launchpad.net/neutron/+bug/1723848</a> is similar and</div><div>corresponding <a href="https://review.openstack.org/#/c/512179/">https://review.openstack.org/#/c/512179/</a> patch was merged in u/s </div><div>and backported to Ocata and pike branches.</div><div><br></div><div>Please also check these [1] relevant patches</div><div>[1] <a href="https://review.openstack.org/#/q/topic:bug/1597461+(status:open+OR+status:merged)">https://review.openstack.org/#/q/topic:bug/1597461+(status:open+OR+status:merged)</a></div><div><br></div><div>Thanks</div><div>Anilvenkata</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 4, 2017 at 6:38 PM, Anna Taraday <span dir="ltr"><<a href="mailto:akamyshnikova@mirantis.com" target="_blank">akamyshnikova@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Please, file a bug about that on <a href="https://bugs.launchpad.net/neutron/+bugs" target="_blank">https://bugs.launchpad.net/<wbr>neutron/+bugs</a> with tag "l3-ha".</div><br><div class="gmail_quote"><div><div class="h5"><div dir="ltr">On Thu, Nov 30, 2017 at 6:00 AM Ajay Kalambur (akalambu) <<a href="mailto:akalambu@cisco.com" target="_blank">akalambu@cisco.com</a>> wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div>
<div>
<div>I noticed that this happens when the router HA interface shows status: Down what can cause the ha interface to do down</div>
<div><br>
</div>
<div>Ajay</div>
<div><br>
</div>
<div>
<div id="m_6193109164415755006m_-1008888023407271136MAC_OUTLOOK_SIGNATURE"></div>
</div>
</div>
</div>
<div><br>
</div>
<span id="m_6193109164415755006m_-1008888023407271136OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri;font-size:12pt;text-align:left;color:black;BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0in;PADDING-LEFT:0in;PADDING-RIGHT:0in;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt">
<span style="font-weight:bold">From: </span>Ajay Kalambur <<a href="mailto:akalambu@cisco.com" target="_blank">akalambu@cisco.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.<wbr>openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, November 29, 2017 at 4:14 PM<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.<wbr>openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>[openstack-dev] [neutron] Neutron L3 agent/Keepalived<br>
</div></span></div><div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif"><span id="m_6193109164415755006m_-1008888023407271136OLK_SRC_BODY_SECTION">
<div><br>
</div>
<span>
<div>
<div style="word-wrap:break-word">
<div>
<div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
Hi</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
I have a case where after running the system for a while I see that floating ip association API gets accepted and no Errors in neutron l3 logs</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
But when I go into the qrouter namespace and check the qg- interface the floating ip is not added</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
Also the keepalived.conf is not updated and SIGUP of the keepalived process is not done</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
So whats the place to look in this case. What is the flow in neutron l3 agent who adds the floating ip to the qg- interface </div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
I see the router update notification received</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
Key log snippet attached. As you can see SIGUP of keepalived is missing and also I confirmed keepalived configs are not updated</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<br>
</div>
<div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:08.883 78 DEBUG oslo_messaging._drivers.<wbr>amqpdriver [-] received message with unique_id: b3886b998c3049e799170bb5351300<wbr>d1 __call__ /usr/lib/python2.7/site-<wbr>packages/oslo_messaging/_<wbr>drivers/amqpdriver.py:269</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:08.885 78 DEBUG neutron.agent.l3.agent [req-6c505d32-2d1a-4392-8ea3-<wbr>0bb888397b9e e759eebecc4648b4964d4ecf439dc0<wbr>ff 13b4f6fb188048ba9bbf211344e334<wbr>2f - - -] Got routers updated notification :[u'a1a59e5f-d74e-404d-a3aa-<wbr>1667c4442aed']
routers_updated /usr/lib/python2.7/site-<wbr>packages/neutron/agent/l3/<wbr>agent.py:413</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:08.886 78 DEBUG neutron.agent.l3.agent [-] Starting router update for a1a59e5f-d74e-404d-a3aa-<wbr>1667c4442aed, action None, priority 0 _process_router_update /usr/lib/python2.7/site-<wbr>packages/neutron/agent/l3/<wbr>agent.py:487</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:08.886 78 DEBUG oslo_messaging._drivers.<wbr>amqpdriver [-] CALL msg_id: 92eb015c8c84489681b1efbe949fab<wbr>8b exchange 'neutron' topic 'q-l3-plugin' _send /usr/lib/python2.7/site-<wbr>packages/oslo_messaging/_<wbr>drivers/amqpdriver.py:568</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.571 78 DEBUG oslo_messaging._drivers.<wbr>amqpdriver [-] received reply msg_id: 92eb015c8c84489681b1efbe949fab<wbr>8b __call__ /usr/lib/python2.7/site-<wbr>packages/oslo_messaging/_<wbr>drivers/amqpdriver.py:416</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.571 78 DEBUG neutron.callbacks.manager [-] Notify callbacks [] for router, before_update _notify_loop /usr/lib/python2.7/site-<wbr>packages/neutron/callbacks/<wbr>manager.py:142</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.572 78 DEBUG neutron.agent.l3.router_info [-] process router updates process /usr/lib/python2.7/site-<wbr>packages/neutron/agent/l3/<wbr>router_info.py:1105</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.572 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-<wbr>a3aa-1667c4442aed', 'find', '/sys/class/net', '-maxdepth', '1', '-type',
'l', '-printf', '%f '] execute_rootwrap_daemon /usr/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>utils.py:105</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.709 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>utils.py:150</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.710 78 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" acquired by "neutron.agent.linux.pd.sync_<wbr>router" :: waited 0.000s inner /usr/lib/python2.7/site-<wbr>packages/oslo_concurrency/<wbr>lockutils.py:270</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.710 78 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" released by "neutron.agent.linux.pd.sync_<wbr>router" :: held 0.000s inner /usr/lib/python2.7/site-<wbr>packages/oslo_concurrency/<wbr>lockutils.py:282</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.710 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-<wbr>a3aa-1667c4442aed', 'find', '/sys/class/net', '-maxdepth', '1', '-type',
'l', '-printf', '%f '] execute_rootwrap_daemon /usr/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>utils.py:105</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.845 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>utils.py:150</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.846 78 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "iptables-qrouter-a1a59e5f-<wbr>d74e-404d-a3aa-1667c4442aed" lock /usr/lib/python2.7/site-<wbr>packages/oslo_concurrency/<wbr>lockutils.py:212</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.846 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-<wbr>a3aa-1667c4442aed', 'iptables-save'] execute_rootwrap_daemon /usr/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>utils.py:105</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.983 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>utils.py:150</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:09.986 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-<wbr>a3aa-1667c4442aed', 'iptables-restore', '-n'] execute_rootwrap_daemon
/usr/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>utils.py:105</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.120 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>utils.py:150</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.120 78 DEBUG neutron.agent.linux.iptables_<wbr>manager [-] IPTablesManager.apply completed with success. 7 iptables commands were issued _apply_synchronized /usr/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>iptables_manager.py:576</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.121 78 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "iptables-qrouter-a1a59e5f-<wbr>d74e-404d-a3aa-1667c4442aed" lock /usr/lib/python2.7/site-<wbr>packages/oslo_concurrency/<wbr>lockutils.py:225</font></div>
<div><font face="Calibri,sans-serif"><b>2017-11-29 14:32:10.121 78 DEBUG neutron.agent.l3.router_info [-] Floating ip 5966c8b3-ff5c-4f01-91a6-<wbr>f81d215bdeca added, status ACTIVE process_floating_ip_addresses /usr/lib/python2.7/site-<wbr>packages/neutron/agent/l3/<wbr>router_info.py:347</b></font></div>
<div><font face="Calibri,sans-serif"><b>2017-11-29 14:32:10.122 78 DEBUG neutron.agent.l3.router_info [-] Sending floating ip statuses: {u'5966c8b3-ff5c-4f01-91a6-<wbr>f81d215bdeca': 'ACTIVE'} update_fip_statuses /usr/lib/python2.7/site-<wbr>packages/neutron/agent/l3/<wbr>router_info.py:915</b></font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.122 78 DEBUG oslo_messaging._drivers.<wbr>amqpdriver [-] CALL msg_id: 18e6ceffec9549a5bd51f65551b655<wbr>96 exchange 'neutron' topic 'q-l3-plugin' _send /usr/lib/python2.7/site-<wbr>packages/oslo_messaging/_<wbr>drivers/amqpdriver.py:568</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.150 78 DEBUG oslo_messaging._drivers.<wbr>amqpdriver [-] received reply msg_id: 18e6ceffec9549a5bd51f65551b655<wbr>96 __call__ /usr/lib/python2.7/site-<wbr>packages/oslo_messaging/_<wbr>drivers/amqpdriver.py:416</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.151 78 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "iptables-qrouter-a1a59e5f-<wbr>d74e-404d-a3aa-1667c4442aed" lock /usr/lib/python2.7/site-<wbr>packages/oslo_concurrency/<wbr>lockutils.py:212</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.151 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-<wbr>a3aa-1667c4442aed', 'iptables-save'] execute_rootwrap_daemon /usr/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>utils.py:105</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.301 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>utils.py:150</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.304 78 DEBUG neutron.agent.linux.iptables_<wbr>manager [-] IPTablesManager.apply completed with success. 0 iptables commands were issued _apply_synchronized /usr/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>iptables_manager.py:576</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.304 78 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "iptables-qrouter-a1a59e5f-<wbr>d74e-404d-a3aa-1667c4442aed" lock /usr/lib/python2.7/site-<wbr>packages/oslo_concurrency/<wbr>lockutils.py:225</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.304 78 DEBUG neutron.callbacks.manager [-] Notify callbacks [('neutron.agent.linux.pd.<wbr>update_router--<wbr>9223372036851355335', <function update_router at 0x3431398>), ('neutron.agent.metadata.<wbr>driver.after_router_updated-<wbr>3681661',
<function after_router_updated at 0x382d7d0>)] for router, after_update _notify_loop /usr/lib/python2.7/site-<wbr>packages/neutron/callbacks/<wbr>manager.py:142</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.305 78 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" acquired by "neutron.agent.linux.pd.<wbr>update_router" :: waited 0.000s inner /usr/lib/python2.7/site-<wbr>packages/oslo_concurrency/<wbr>lockutils.py:270</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.305 78 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" released by "neutron.agent.linux.pd.<wbr>update_router" :: held 0.000s inner /usr/lib/python2.7/site-<wbr>packages/oslo_concurrency/<wbr>lockutils.py:282</font></div>
<div><font face="Calibri,sans-serif">2017-11-29 14:32:10.305 78 DEBUG neutron.agent.l3.agent [-] Finished a router update for a1a59e5f-d74e-404d-a3aa-<wbr>1667c4442aed _process_router_update /usr/lib/python2.7/site-<wbr>packages/neutron/agent/l3/<wbr>agent.py:536</font></div>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<div id="m_6193109164415755006m_-1008888023407271136"></div>
</div>
</div>
</div>
</div>
</div>
</span></span></div></div></div>
______________________________<wbr>______________________________<wbr>______________<span class=""><br>
OpenStack Development Mailing List (not for usage questions)<br></span>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><span class="HOEnZb"><font color="#888888"><br>
</font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><div dir="ltr">-- <br></div><div class="m_6193109164415755006gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Regards,<br>Ann Taraday</div></div>
</font></span><br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br></blockquote></div><br></div>