[openstack-dev] [Neutron][LBaaS] LBaaSv2 with HAproxy Agent Deployment Issue

Daneyon Hansen (danehans) danehans at cisco.com
Wed Jun 8 04:07:53 UTC 2016


All,

I am trying to add Neutron LBaaSv2 to a working OpenStack Liberty deployment. I am running into an issue where the lbaas agent does not appear in the output of neutron agent-list. However, the lbaas extension appears in the output of neutron ext-list. After investigating further, the lbaas-agent sends a message on the queue and times out waiting for a reply:

2016-06-06 21:09:15.958 22 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.32.20.52:5672
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager [-] Unable to retrieve ready devices
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager Traceback (most recent call last):
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutro
n_lbaas/services/loadbalancer/agent/agent_manager.py", line 152, in sync_state
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     ready_instances = set(self.plugin_rpc.get_r
eady_devices())
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutro
n_lbaas/services/loadbalancer/agent/agent_api.py", line 36, in get_ready_devices
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     return cctxt.call(self.context, 'get_ready_
devices', host=self.host)
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_m
essaging/rpc/client.py", line 158, in call
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     retry=self.retry)
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_m
essaging/transport.py", line 90, in _send
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     timeout=timeout, retry=retry)
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_m
essaging/_drivers/amqpdriver.py", line 431, in send
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     retry=retry)
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_m
essaging/_drivers/amqpdriver.py", line 420, in _send
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     result = self._waiter.wait(msg_id, timeout)
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_m
essaging/_drivers/amqpdriver.py", line 318, in wait
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     message = self.waiters.get(msg_id, timeout=
timeout)
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_m
essaging/_drivers/amqpdriver.py", line 223, in get
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     'to message ID %s' % msg_id)
2016-06-06 21:10:15.972 22 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager MessagingTimeout: Timed out waiting for a reply
 to message ID eae3cc1bc8614aa8ae499d92ca4ec731

I verified that the lbaas queues reside within the Rabbit cluster:

bash-4.2$ rabbitmqctl list_queues

n-lbaas_agent   0
n-lbaas_agent.control-server-1.novalocal        0
n-lbaas_agent.control-server-2.novalocal        0
n-lbaas_agent.control-server-3.novalocal        0
n-lbaas_agent_fanout_18a3b28c969148f3a008df8f3e5f5363   0
n-lbaas_agent_fanout_a7d48e8a1b27443d82ee4944bec44cf8   0
n-lbaas_agent_fanout_b5360edb19c240e79c71d60806977f66   0
n-lbaasv2-plugin        0
n-lbaasv2-plugin.control-server-1.novalocal     0
n-lbaasv2-plugin.control-server-2.novalocal     0
n-lbaasv2-plugin.control-server-3.novalocal     0
n-lbaasv2-plugin_fanout_5cbb6dd4fafc4c4784add8a20e0a28a5        0
n-lbaasv2-plugin_fanout_756ee4e4eee547528d0f6e3dde71b150        0
n-lbaasv2-plugin_fanout_7629f7bb85ce493d83c334dfcc2cd4aa        0
notifications.info      8


And the lbaas queues are being mirrored:

# rabbitmq server logs
=INFO REPORT==== 6-Jun-2016::19:01:23 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_659b460849ef43ee834ce6d88d294b46' in vhost '/': Adding mirror on node 'rabbit at mercury-control-se
rver-3': <3038.25481.1>

=INFO REPORT==== 6-Jun-2016::19:01:23 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_659b460849ef43ee834ce6d88d294b46' in vhost '/': Adding mirror on node 'rabbit at mercury-control-se
rver-2': <3037.25635.1>

=INFO REPORT==== 6-Jun-2016::19:01:23 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_659b460849ef43ee834ce6d88d294b46' in vhost '/': Synchronising: 0 messages to synchronise

=INFO REPORT==== 6-Jun-2016::19:01:23 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_659b460849ef43ee834ce6d88d294b46' in vhost '/': Synchronising: all slaves already synced

=INFO REPORT==== 6-Jun-2016::19:01:23 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_659b460849ef43ee834ce6d88d294b46' in vhost '/': Synchronising: 0 messages to synchronise

=INFO REPORT==== 6-Jun-2016::19:01:23 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_659b460849ef43ee834ce6d88d294b46' in vhost '/': Synchronising: all slaves already synced
...
=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Adding mirror on node 'rabbit at mercury-control-se
rver-3': <3038.25503.1>

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Adding mirror on node 'rabbit at mercury-control-se
rver-2': <3037.25657.1>

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: 0 messages to synchronise

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Adding mirror on node 'rabbit at mercury-control-se
rver-3': <3038.25506.1>

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: all slaves already synced

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: 0 messages to synchronise

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: all slaves already synced

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Adding mirror on node 'rabbit at mercury-control-se
rver-2': <3037.25660.1>

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: 0 messages to synchronise

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_a6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: all slaves already synced

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Adding mirror on node 'rabbit at mercury-control-se
rver-2': <3037.25660.1>

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Synchronising: 0 messages to synchronise

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Synchronising: all slaves already synced

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Synchronising: 0 messages to synchronise

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-plugin_fanout_3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Synchronising: all slaves already synced


All other neutron agents and openstack services are functional and communicate across the same Rabbit cluster. [1] provides additional details of the setup. It appears I may be running into bug [2]. I added myself to this bug, but it seems to have gone dormant. I would appreciate any support in resolving this issue.

[1] https://gist.github.com/danehans/8691464051cb43feacac11afd4bbc447
[2] https://bugs.launchpad.net/neutron/+bug/1464241

Regards,
Daneyon Hansen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160608/d6405efd/attachment.html>


More information about the OpenStack-dev mailing list