[nova][neutron][oslo][ops] rabbit bindings issue

Fabian Zimmermann dev.faz at gmail.com
Sat Aug 8 04:30:09 UTC 2020


Hi,

we also have this issue.

Our solution was (up to now) to delete the queues with a script or even
reset the complete cluster.

We just upgraded rabbitmq to the latest version - without luck.

Anyone else seeing this issue?

 Fabian



Arnaud Morin <arnaud.morin at gmail.com> schrieb am Do., 6. Aug. 2020, 16:47:

> Hey all,
>
> I would like to ask the community about a rabbit issue we have from time
> to time.
>
> In our current architecture, we have a cluster of rabbits (3 nodes) for
> all our OpenStack services (mostly nova and neutron).
>
> When one node of this cluster is down, the cluster continue working (we
> use pause_minority strategy).
> But, sometimes, the third server is not able to recover automatically
> and need a manual intervention.
> After this intervention, we restart the rabbitmq-server process, which
> is then able to join the cluster back.
>
> At this time, the cluster looks ok, everything is fine.
> BUT, nothing works.
> Neutron and nova agents are not able to report back to servers.
> They appear dead.
> Servers seems not being able to consume messages.
> The exchanges, queues, bindings seems good in rabbit.
>
> What we see is that removing bindings (using rabbitmqadmin delete
> binding or the web interface) and recreate them again (using the same
> routing key) brings the service back up and running.
>
> Doing this for all queues is really painful. Our next plan is to
> automate it, but is there anyone in the community already saw this kind
> of issues?
>
> Our bug looks like the one described in [1].
> Someone recommands to create an Alternate Exchange.
> Is there anyone already tried that?
>
> FYI, we are running rabbit 3.8.2 (with OpenStack Stein).
> We had the same kind of issues using older version of rabbit.
>
> Thanks for your help.
>
> [1]
> https://groups.google.com/forum/#!newtopic/rabbitmq-users/rabbitmq-users/zFhmpHF2aWk
>
> --
> Arnaud Morin
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200808/ed7f0bca/attachment.html>


More information about the openstack-discuss mailing list