<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="color: rgb(0, 0, 0); font-size: 14px;">All,</div>
<div style="color: rgb(0, 0, 0); font-size: 14px;"><br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px;">I am trying to add Neutron LBaaSv2 to a working OpenStack Liberty deployment. I am running into an issue where the lbaas agent does not appear in the output of neutron agent-list. However, the lbaas extension
appears in the output of neutron ext-list. After investigating further, the lbaas-agent sends a message on the queue and times out waiting for a reply:</div>
<div style="color: rgb(0, 0, 0); font-size: 14px;">
<pre style="widows: 1; word-wrap: break-word; white-space: pre-wrap;"><font face="Calibri">2016-06-06 21:09:15.958 22 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.32.20.52:5672
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager [-] Unable to retrieve ready devices
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager Traceback (most recent call last):
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutro
n_lbaas/services/loadbalancer/agent/agent_manager.py", line 152, in sync_state
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager ready_instances = set(self.plugin_rpc.get_r
eady_devices())
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutro
n_lbaas/services/loadbalancer/agent/agent_api.py", line 36, in get_ready_devices
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager return cctxt.call(self.context, 'get_ready_
devices', host=self.host)
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_m
essaging/rpc/client.py", line 158, in call
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager retry=self.retry)
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_m
essaging/transport.py", line 90, in _send
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager timeout=timeout, retry=retry)
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_m
essaging/_drivers/amqpdriver.py", line 431, in send
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager retry=retry)
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_m
essaging/_drivers/amqpdriver.py", line 420, in _send
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager result = self._waiter.wait(msg_id, timeout)
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_m
essaging/_drivers/amqpdriver.py", line 318, in wait
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager message = self.waiters.get(msg_id, timeout=
timeout)
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_m
essaging/_drivers/amqpdriver.py", line 223, in get
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 'to message ID %s' % msg_id)
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager MessagingTimeout: Timed out waiting for a reply
to message ID eae3cc1bc8614aa8ae499d92ca4ec731</font></pre>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px;"><br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px;">I verified that the lbaas queues reside within the Rabbit cluster:</div>
<div>
<pre style="color: rgb(0, 0, 0); font-size: 14px; widows: 1; word-wrap: break-word; white-space: pre-wrap;"><font face="Calibri">bash-4.2$ rabbitmqctl list_queues</font></pre>
<pre style="color: rgb(0, 0, 0); font-size: 14px; widows: 1; word-wrap: break-word; white-space: pre-wrap;"><pre style="word-wrap: break-word; white-space: pre-wrap;"><font face="Calibri">n-lbaas_agent 0
n-lbaas_agent.control-server-1.novalocal 0
n-lbaas_agent.control-server-2.novalocal 0
n-lbaas_agent.control-server-3.novalocal 0
n-lbaas_agent_fanout_18a3b28c969148f3a008df8f3e5f5363 0
n-lbaas_agent_fanout_a7d48e8a1b27443d82ee4944bec44cf8 0
n-lbaas_agent_fanout_b5360edb19c240e79c71d60806977f66 0
n-lbaasv2-plugin 0
n-lbaasv2-plugin.control-server-1.novalocal 0
n-lbaasv2-plugin.control-server-2.novalocal 0
n-lbaasv2-plugin.control-server-3.novalocal 0
n-lbaasv2-plugin_fanout_5cbb6dd4fafc4c4784add8a20e0a28a5 0
n-lbaasv2-plugin_fanout_756ee4e4eee547528d0f6e3dde71b150 0
n-lbaasv2-plugin_fanout_7629f7bb85ce493d83c334dfcc2cd4aa 0
notifications.info 8</font></pre></pre>
<pre style="color: rgb(0, 0, 0); font-size: 14px; widows: 1; word-wrap: break-word; white-space: pre-wrap;"><font face="Calibri"><br></font></pre>
<pre style="color: rgb(0, 0, 0); font-size: 14px; widows: 1; word-wrap: break-word; white-space: pre-wrap;"><font face="Calibri">And the lbaas queues are being mirrored:</font></pre>
<pre style="widows: 1; word-wrap: break-word;"><pre style="color: rgb(0, 0, 0); font-size: 14px; white-space: pre-wrap; word-wrap: break-word;"><font face="Calibri"># rabbitmq server logs
=INFO REPORT==== 6-Jun-2016::19:01:23 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_659b460849ef43ee834ce6d88d294b46' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se
rver-3': <3038.25481.1>
=INFO REPORT==== 6-Jun-2016::19:01:23 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_659b460849ef43ee834ce6d88d294b46' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se
rver-2': <3037.25635.1>
=INFO REPORT==== 6-Jun-2016::19:01:23 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_659b460849ef43ee834ce6d88d294b46' in vhost '/': Synchronising: 0 messages to synchronise
=INFO REPORT==== 6-Jun-2016::19:01:23 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_659b460849ef43ee834ce6d88d294b46' in vhost '/': Synchronising: all slaves already synced
=INFO REPORT==== 6-Jun-2016::19:01:23 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_659b460849ef43ee834ce6d88d294b46' in vhost '/': Synchronising: 0 messages to synchronise
=INFO REPORT==== 6-Jun-2016::19:01:23 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_659b460849ef43ee834ce6d88d294b46' in vhost '/': Synchronising: all slaves already synced
...
=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se
rver-3': <3038.25503.1>
=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se
rver-2': <3037.25657.1>
=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: 0 messages to synchronise
=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se
rver-3': <3038.25506.1>
=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: all slaves already synced
=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: 0 messages to synchronise
=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: all slaves already synced
=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se
rver-2': <3037.25660.1>
=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: 0 messages to synchronise
=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: all slaves already synced
=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se
rver-2': <3037.25660.1>
=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Synchronising: 0 messages to synchronise
=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Synchronising: all slaves already synced
=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Synchronising: 0 messages to synchronise
=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Synchronising: all slaves already synced</font></pre><pre style="color: rgb(0, 0, 0); font-size: 14px; white-space: pre-wrap; word-wrap: break-word;"><font face="Calibri"><br></font></pre><pre style="word-wrap: break-word;"><font face="Calibri"><span style="white-space: pre-wrap;">All other neutron agents and openstack services are functional and communicate across the same Rabbit cluster. </span><span style="white-space: pre-wrap;">[1] provides additional details of the setup. It appears I may be running into bug [2]. I added myself to this bug, but it seems to have gone dormant. I would appreciate any support in resolving this issue.</span></font></pre></pre>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px;">[1] <a href="https://gist.github.com/danehans/8691464051cb43feacac11afd4bbc447">https://gist.github.com/danehans/8691464051cb43feacac11afd4bbc447</a></div>
<div style="color: rgb(0, 0, 0); font-size: 14px;">[2] <a href="https://bugs.launchpad.net/neutron/+bug/1464241">https://bugs.launchpad.net/neutron/+bug/1464241</a></div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>Regards,</div>
<div>Daneyon Hansen</div>
<div></div>
</div>
</body>
</html>