Open Stack

Hi
I have a case where after running the system for a while I see that floating ip association API gets accepted and no Errors in neutron l3 logs
But when I go into the qrouter namespace and check the qg- interface the floating ip is not added
Also the keepalived.conf is not updated and SIGUP of the keepalived process is not done

So whats the place to look in this case.  What is the flow in neutron l3 agent who adds the floating ip to the qg- interface
I see the router update notification received
Key log snippet attached. As you can see SIGUP of keepalived is missing and also I confirmed keepalived configs are not updated


2017-11-29 14:32:08.883 78 DEBUG oslo_messaging._drivers.amqpdriver [-] received message with unique_id: b3886b998c3049e799170bb5351300d1 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:269
2017-11-29 14:32:08.885 78 DEBUG neutron.agent.l3.agent [req-6c505d32-2d1a-4392-8ea3-0bb888397b9e e759eebecc4648b4964d4ecf439dc0ff 13b4f6fb188048ba9bbf211344e3342f - - -] Got routers updated notification :[u'a1a59e5f-d74e-404d-a3aa-1667c4442aed'] routers_updated /usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:413
2017-11-29 14:32:08.886 78 DEBUG neutron.agent.l3.agent [-] Starting router update for a1a59e5f-d74e-404d-a3aa-1667c4442aed, action None, priority 0 _process_router_update /usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:487
2017-11-29 14:32:08.886 78 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 92eb015c8c84489681b1efbe949fab8b exchange 'neutron' topic 'q-l3-plugin' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:568

2017-11-29 14:32:09.571 78 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 92eb015c8c84489681b1efbe949fab8b __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:416
2017-11-29 14:32:09.571 78 DEBUG neutron.callbacks.manager [-] Notify callbacks [] for router, before_update _notify_loop /usr/lib/python2.7/site-packages/neutron/callbacks/manager.py:142
2017-11-29 14:32:09.572 78 DEBUG neutron.agent.l3.router_info [-] process router updates process /usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:1105
2017-11-29 14:32:09.572 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-11-29 14:32:09.709 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-11-29 14:32:09.710 78 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" acquired by "neutron.agent.linux.pd.sync_router" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-11-29 14:32:09.710 78 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" released by "neutron.agent.linux.pd.sync_router" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-11-29 14:32:09.710 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-11-29 14:32:09.845 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-11-29 14:32:09.846 78 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "iptables-qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2017-11-29 14:32:09.846 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'iptables-save'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-11-29 14:32:09.983 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-11-29 14:32:09.986 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'iptables-restore', '-n'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-11-29 14:32:10.120 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-11-29 14:32:10.120 78 DEBUG neutron.agent.linux.iptables_manager [-] IPTablesManager.apply completed with success. 7 iptables commands were issued _apply_synchronized /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:576
2017-11-29 14:32:10.121 78 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "iptables-qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225
2017-11-29 14:32:10.121 78 DEBUG neutron.agent.l3.router_info [-] Floating ip 5966c8b3-ff5c-4f01-91a6-f81d215bdeca added, status ACTIVE process_floating_ip_addresses /usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:347
2017-11-29 14:32:10.122 78 DEBUG neutron.agent.l3.router_info [-] Sending floating ip statuses: {u'5966c8b3-ff5c-4f01-91a6-f81d215bdeca': 'ACTIVE'} update_fip_statuses /usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:915
2017-11-29 14:32:10.122 78 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 18e6ceffec9549a5bd51f65551b65596 exchange 'neutron' topic 'q-l3-plugin' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:568
2017-11-29 14:32:10.150 78 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 18e6ceffec9549a5bd51f65551b65596 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:416
2017-11-29 14:32:10.151 78 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "iptables-qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2017-11-29 14:32:10.151 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'iptables-save'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-11-29 14:32:10.301 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-11-29 14:32:10.304 78 DEBUG neutron.agent.linux.iptables_manager [-] IPTablesManager.apply completed with success. 0 iptables commands were issued _apply_synchronized /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:576
2017-11-29 14:32:10.304 78 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "iptables-qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225
2017-11-29 14:32:10.304 78 DEBUG neutron.callbacks.manager [-] Notify callbacks [('neutron.agent.linux.pd.update_router--9223372036851355335', <function update_router at 0x3431398>), ('neutron.agent.metadata.driver.after_router_updated-3681661', <function after_router_updated at 0x382d7d0>)] for router, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron/callbacks/manager.py:142
2017-11-29 14:32:10.305 78 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" acquired by "neutron.agent.linux.pd.update_router" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-11-29 14:32:10.305 78 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" released by "neutron.agent.linux.pd.update_router" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-11-29 14:32:10.305 78 DEBUG neutron.agent.l3.agent [-] Finished a router update for a1a59e5f-d74e-404d-a3aa-1667c4442aed _process_router_update /usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:536
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20171130/1283c900/attachment.html>