Open Stack

Hello,
Thank you guys for your links. Actually I moved from no durable queues + no
HA policy to  durable queues + ha-all policy. The result is still the same.
Tried to turning using
https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit but still
missing something I guess.
@Albert: Have you tested the case when you shutdown 1 controller -> thing
works -> power it on -> shutdown another controller? In my case the cluster
is not stable after that.
And by "work fine" you mean you don't have to do anything (restart
rabbitmq, restart openstack services) when 1 controller is down, do you? I
know it sounds silly, but we end up using internal keepalived VIP only for
all transport settings which remove loadbalancing but keep my cluster
stable when 1 node down, really don't know if it will cause trouble later
when the cluster grows.

Regards,
Tan


On Fri, Jul 22, 2022 at 10:53 PM Albert Braden <ozzzo at yahoo.com> wrote:

> The default RMQ config is broken. You're on the right track with setting
> durable_queues, but there's more to do. I'm running kolla Train with
> mirrored/durable queues and my clusters work fine with a controller down.
> One issue that we faced after setting durable was that we weren't running
> redis, and then when we tried to run it the network was blocking the port,
> but eventually we got it working.
>
> Some have recommended not mirroring queues; I haven't tried that. If
> anyone has successfully setup HA without mirrored queues, I'd be interested
> to hear about how you did it.
>
> Here are some helpful links:
>
> https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit
>
> https://lists.openstack.org/pipermail/openstack-discuss/2021-November/026074.html
>
> https://lists.openstack.org/pipermail/openstack-discuss/2020-August/016362.html
>
> https://lists.openstack.org/pipermail/openstack-discuss/2020-August/016524.html
> https://review.opendev.org/c/openstack/kolla-ansible/+/822191
> https://review.opendev.org/c/openstack/kolla-ansible/+/824994
> On Thursday, July 21, 2022, 02:42:42 PM EDT, Tan Tran Trong <
> gk.coltech at gmail.com> wrote:
>
>
> Hello,
> I'm trying to figure out how to configure RabbitMQ to make it high
> available. I have 3 controller nodes and 2 compute nodes, deployed with
> kolla with mostly default configuration. The RabbitMQ set to ha-all for all
> queues on all nodes, amqp_durable_queues = True
> My problem is when I shutdown 1 controller node (or 1 RabbitMQ container)
> (master or slave) the whole cluster becomes unstable. Some instances can
> not be created, it is stuck on Scheduling, Block Device Mapping, the
> volumes not shown or are stuck on creating, the compute node reported dead
> randomly,...
> I'm looking for documentation to know how Openstack using RabbitMQ,
> Openstack behavior when RabbitMQ node down and way to make RabbitMQ HA in a
> stable way. Do you have any recommendation?
>
> TIA,
> Tan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20220724/e511c9e0/attachment.htm>