[rabbitmq] Need help about RabbitMQ quorum HA mode
Hi, In kolla-ansible yoga environment, rabbitmq queues are set to quorum by default instead of classical mirroring, as it seems they are not replicated (per queue has +0 quorum). We tried to find a solution to handle it in configs, but we couldn't find it and we set queues to be replicated by using CLI commands (rabbitmq-queues grow). Now, my questions are: 1- Why do all rabbitmq queues come as non-replicated by default? 2- How can we set HA mode to quorum queues?
Hi, Quorum queues are default since Kolla-Ansible 2023.3 Bobcat release. There is a way to upgrade to quorum queues if you switch to a newer OpenStack release but I don’t think this has been backported to yoga. The Yoga release is EOL. Maybe there is some hacky way to get it working. Cheers, Oliver Von meinem iPhone gesendet
Am 02.01.2025 um 12:19 schrieb İzzettin Erdem <root.mch@gmail.com>:
Hi,
In kolla-ansible yoga environment, rabbitmq queues are set to quorum by default instead of classical mirroring, as it seems they are not replicated (per queue has +0 quorum).
We tried to find a solution to handle it in configs, but we couldn't find it and we set queues to be replicated by using CLI commands (rabbitmq-queues grow).
Now, my questions are: 1- Why do all rabbitmq queues come as non-replicated by default? 2- How can we set HA mode to quorum queues?
Hello! Given you're using Yoga, the Kolla-Ansible option you're looking for is om_enable_rabbitmq_high_availability. This will enable durable queues and classic queue mirroring. Please see the Yoga KA docs for info on how you can migrate to these in an existing environment: https://docs.openstack.org/kolla-ansible/yoga/reference/message-queues/rabbi... On quorum queues, I'm afraid KA don't support these in Yoga. Support was added in 2023.1, and then we enabled quorum queues by default in 2024.1 You would need to upgrade to 2023.1 to use these via om_enable_rabbitmq_quorum_queues. (2023.1 docs: https://docs.openstack.org/kolla-ansible/2023.1/reference/message-queues/rab... ) Hope this helps, Matt Crees On Thu, 2 Jan 2025 at 11:19, İzzettin Erdem <root.mch@gmail.com> wrote:
Hi,
In kolla-ansible yoga environment, rabbitmq queues are set to quorum by default instead of classical mirroring, as it seems they are not replicated (per queue has +0 quorum).
We tried to find a solution to handle it in configs, but we couldn't find it and we set queues to be replicated by using CLI commands (rabbitmq-queues grow).
Now, my questions are: 1- Why do all rabbitmq queues come as non-replicated by default? 2- How can we set HA mode to quorum queues?
Hi Matt, Thanks for your reply, I wrote it wrong, I was going to write caracal (2024.1) instead of yoga, sorry for that. We added 2 more controller nodes to this environment (total 5) and deployed kolla-ansible from scratch. We put these parameters into globals.yaml, but it did not work. Some of the queues have 1 replica, some of them have 2 or 0 (quorum queues, not classic mirroring). We want to set all queues to be replicated because if the leader node goes down, all services go down. We had an incident at production, still trying to find a way to fix this situation. İzzettin Matt Crees <mattc@stackhpc.com>, 2 Oca 2025 Per, 14:36 tarihinde şunu yazdı:
Hello!
Given you're using Yoga, the Kolla-Ansible option you're looking for is om_enable_rabbitmq_high_availability. This will enable durable queues and classic queue mirroring. Please see the Yoga KA docs for info on how you can migrate to these in an existing environment: https://docs.openstack.org/kolla-ansible/yoga/reference/message-queues/rabbi...
On quorum queues, I'm afraid KA don't support these in Yoga. Support was added in 2023.1, and then we enabled quorum queues by default in 2024.1 You would need to upgrade to 2023.1 to use these via om_enable_rabbitmq_quorum_queues. (2023.1 docs: https://docs.openstack.org/kolla-ansible/2023.1/reference/message-queues/rab... )
Hope this helps, Matt Crees
On Thu, 2 Jan 2025 at 11:19, İzzettin Erdem <root.mch@gmail.com> wrote:
Hi,
In kolla-ansible yoga environment, rabbitmq queues are set to quorum by default instead of classical mirroring, as it seems they are not replicated (per queue has +0 quorum).
We tried to find a solution to handle it in configs, but we couldn't find it and we set queues to be replicated by using CLI commands (rabbitmq-queues grow).
Now, my questions are: 1- Why do all rabbitmq queues come as non-replicated by default? 2- How can we set HA mode to quorum queues?
Hello, We are not using kolla but we exclusively use quorum queues in our deployments. We monitor the number of replica and the fact that a quorum queue is healthy using the command: rabbitmq-queues check_if_node_is_quorum_critical (see [1]) This may help you find out which queues are not in good health. What you also need to do is to add_member for the queue (same cli tool) if it's not replicated correctly (e.g. you remove / add a new rabbitmq node). AFAIR, re-election of leaders is not trigger automatically. Note that if you are using stream queues, you may need to perform similar action using rabbitmq-stream cli tool. [1] https://www.rabbitmq.com/docs/man/rabbitmq-queues.8#check_if_node_is_quorum_... Regards, Arnaud. On 02.01.25 - 15:23, İzzettin Erdem wrote:
Hi Matt,
Thanks for your reply, I wrote it wrong, I was going to write caracal (2024.1) instead of yoga, sorry for that.
We added 2 more controller nodes to this environment (total 5) and deployed kolla-ansible from scratch. We put these parameters into globals.yaml, but it did not work. Some of the queues have 1 replica, some of them have 2 or 0 (quorum queues, not classic mirroring).
We want to set all queues to be replicated because if the leader node goes down, all services go down. We had an incident at production, still trying to find a way to fix this situation.
İzzettin
Matt Crees <mattc@stackhpc.com>, 2 Oca 2025 Per, 14:36 tarihinde şunu yazdı:
Hello!
Given you're using Yoga, the Kolla-Ansible option you're looking for is om_enable_rabbitmq_high_availability. This will enable durable queues and classic queue mirroring. Please see the Yoga KA docs for info on how you can migrate to these in an existing environment: https://docs.openstack.org/kolla-ansible/yoga/reference/message-queues/rabbi...
On quorum queues, I'm afraid KA don't support these in Yoga. Support was added in 2023.1, and then we enabled quorum queues by default in 2024.1 You would need to upgrade to 2023.1 to use these via om_enable_rabbitmq_quorum_queues. (2023.1 docs: https://docs.openstack.org/kolla-ansible/2023.1/reference/message-queues/rab... )
Hope this helps, Matt Crees
On Thu, 2 Jan 2025 at 11:19, İzzettin Erdem <root.mch@gmail.com> wrote:
Hi,
In kolla-ansible yoga environment, rabbitmq queues are set to quorum by default instead of classical mirroring, as it seems they are not replicated (per queue has +0 quorum).
We tried to find a solution to handle it in configs, but we couldn't find it and we set queues to be replicated by using CLI commands (rabbitmq-queues grow).
Now, my questions are: 1- Why do all rabbitmq queues come as non-replicated by default? 2- How can we set HA mode to quorum queues?
Just to have that said - I don't think we handle or verify quorum state right now in OpenStack-Ansible either. While improving flow for re-bootstrapping/growing/shrinking the cluster with quorum queues is totally in the list - it is not implemented so far. So operators are encouraged to check the state of queues with `rabbitmq-queues check_if_node_is_quorum_critical` before doing maintenance at the moment. Hopefully, we will be able to work on required fixes for the Epoxy. вс, 12 янв. 2025 г. в 21:19, Arnaud Morin <arnaud.morin@gmail.com>:
Hello,
We are not using kolla but we exclusively use quorum queues in our deployments.
We monitor the number of replica and the fact that a quorum queue is healthy using the command: rabbitmq-queues check_if_node_is_quorum_critical (see [1])
This may help you find out which queues are not in good health.
What you also need to do is to add_member for the queue (same cli tool) if it's not replicated correctly (e.g. you remove / add a new rabbitmq node). AFAIR, re-election of leaders is not trigger automatically.
Note that if you are using stream queues, you may need to perform similar action using rabbitmq-stream cli tool.
[1] https://www.rabbitmq.com/docs/man/rabbitmq-queues.8#check_if_node_is_quorum_...
Regards, Arnaud.
On 02.01.25 - 15:23, İzzettin Erdem wrote:
Hi Matt,
Thanks for your reply, I wrote it wrong, I was going to write caracal (2024.1) instead of yoga, sorry for that.
We added 2 more controller nodes to this environment (total 5) and deployed kolla-ansible from scratch. We put these parameters into globals.yaml, but it did not work. Some of the queues have 1 replica, some of them have 2 or 0 (quorum queues, not classic mirroring).
We want to set all queues to be replicated because if the leader node goes down, all services go down. We had an incident at production, still trying to find a way to fix this situation.
İzzettin
Matt Crees <mattc@stackhpc.com>, 2 Oca 2025 Per, 14:36 tarihinde şunu yazdı:
Hello!
Given you're using Yoga, the Kolla-Ansible option you're looking for is om_enable_rabbitmq_high_availability. This will enable durable queues and classic queue mirroring. Please see the Yoga KA docs for info on how you can migrate to these in an existing environment: https://docs.openstack.org/kolla-ansible/yoga/reference/message-queues/rabbi...
On quorum queues, I'm afraid KA don't support these in Yoga. Support was added in 2023.1, and then we enabled quorum queues by default in 2024.1 You would need to upgrade to 2023.1 to use these via om_enable_rabbitmq_quorum_queues. (2023.1 docs: https://docs.openstack.org/kolla-ansible/2023.1/reference/message-queues/rab... )
Hope this helps, Matt Crees
On Thu, 2 Jan 2025 at 11:19, İzzettin Erdem <root.mch@gmail.com> wrote:
Hi,
In kolla-ansible yoga environment, rabbitmq queues are set to quorum by default instead of classical mirroring, as it seems they are not replicated (per queue has +0 quorum).
We tried to find a solution to handle it in configs, but we couldn't find it and we set queues to be replicated by using CLI commands (rabbitmq-queues grow).
Now, my questions are: 1- Why do all rabbitmq queues come as non-replicated by default? 2- How can we set HA mode to quorum queues?
participants (5)
-
Arnaud Morin
-
Dmitriy Rabotyagov
-
İzzettin Erdem
-
Matt Crees
-
Oliver Weinmann