Hello Huettner, I have used the quorum queue since March and it is ok until now. It looks more stable than the classic queue. Some feedback to you. Thank you. Nguyen Huu Khoi. On Mon, May 8, 2023 at 1:14 PM Felix Hüttner <felix.huettner@mail.schwarz> wrote:

Hi Nguyen,

we are using quorum queues for one of our deployments. So fare we did not have any issue with them. They also seem to survive restarts without issues (however reply queues are still broken afterwards in a small amount of cases, but they are no quorum/mirrored queues anyway).

So I would recommend them for everyone that creates a new cluster.

--

Felix Huettner

*From:* Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> *Sent:* Saturday, May 6, 2023 4:29 AM *To:* OpenStack Discuss <openstack-discuss@lists.openstack.org> *Subject:* [OPENSTACK][rabbitmq] using quorum queues

Hello guys.

IS there any guy who uses the quorum queue for openstack? Could you give some feedback to compare with classic queue?

Thank you.

Nguyen Huu Khoi

Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.

Hinweise zum Datenschutz finden Sie hier <https://www.datenschutz.schwarz> .

This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.

Information on data protection can be found here <https://www.datenschutz.schwarz>.

Hello. Firstly, when I used the classic queue and sometimes, my rabbitmq cluster was broken, the computers showed state down and I needed to restart the computer service to make it up. Secondly, 1 of 3 controller is down but my system still works although it is not very first as fully controller. I ran it for about 3 months compared with classic. My openstack is Yoga and use Kolla-Ansible as a deployment tool, Nguyen Huu Khoi On Tue, Jun 13, 2023 at 8:43 AM Sa Pham <saphi070@gmail.com> wrote:

Hi Khôi,

Why do you say using the quorum queue is more stable than the classic queue ?

Thanks,

On Tue, Jun 13, 2023 at 7:26 AM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:

...

Hello Huettner, I have used the quorum queue since March and it is ok until now. It looks more stable than the classic queue. Some feedback to you. Thank you. Nguyen Huu Khoi.

On Mon, May 8, 2023 at 1:14 PM Felix Hüttner <felix.huettner@mail.schwarz> wrote:

...

Hi Nguyen,

we are using quorum queues for one of our deployments. So fare we did not have any issue with them. They also seem to survive restarts without issues (however reply queues are still broken afterwards in a small amount of cases, but they are no quorum/mirrored queues anyway).

So I would recommend them for everyone that creates a new cluster.

--

Felix Huettner

*From:* Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> *Sent:* Saturday, May 6, 2023 4:29 AM *To:* OpenStack Discuss <openstack-discuss@lists.openstack.org> *Subject:* [OPENSTACK][rabbitmq] using quorum queues

Hello guys.

IS there any guy who uses the quorum queue for openstack? Could you give some feedback to compare with classic queue?

Thank you.

Nguyen Huu Khoi

Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.

Hinweise zum Datenschutz finden Sie hier <https://www.datenschutz.schwarz>.

This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.

Information on data protection can be found here <https://www.datenschutz.schwarz>.

-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582

Hey, We are also using quorum in some regions and plan to enable quorum everwhere. Note that we also manage to enable quorum for transient queues (using a custom patch as it's not doable with current oslo.messaging, see my request in [1]). We also introduced some custom changes in py-amqp to handle correctly the rabbit disconnections (see [2] and [3]). So far, the real improvment is achieved thanks to the combination of all of these changes, enabling quorum queue only was not enough for us to notice any improvment. The downside of quorum queues is that it consume more power on the rabbit cluster: you need more IO, CPU, RAM and network bandwith for the same number of queues (see [4]). It has to be taken into account. Cheers, Arnaud. [1] https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033343.ht... [2] https://github.com/celery/py-amqp/pull/410 [3] https://github.com/celery/py-amqp/pull/405 [4] https://plik.ovh/file/nHCny7psDCTrEm76/Pq9ASO9wUd8HRk4C/s_1686817000.png On 13.06.23 - 09:14, Sa Pham wrote:

Dear Khôi,

Thanks for your reply.

On Tue, Jun 13, 2023 at 9:05 AM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:

...

Hello. Firstly, when I used the classic queue and sometimes, my rabbitmq cluster was broken, the computers showed state down and I needed to restart the computer service to make it up. Secondly, 1 of 3 controller is down but my system still works although it is not very first as fully controller. I ran it for about 3 months compared with classic. My openstack is Yoga and use Kolla-Ansible as a deployment tool, Nguyen Huu Khoi

On Tue, Jun 13, 2023 at 8:43 AM Sa Pham <saphi070@gmail.com> wrote:

...

Hi Khôi,

Why do you say using the quorum queue is more stable than the classic queue ?

Thanks,

On Tue, Jun 13, 2023 at 7:26 AM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:

...

Hello Huettner, I have used the quorum queue since March and it is ok until now. It looks more stable than the classic queue. Some feedback to you. Thank you. Nguyen Huu Khoi.

On Mon, May 8, 2023 at 1:14 PM Felix Hüttner <felix.huettner@mail.schwarz> wrote:

...

Hi Nguyen,

we are using quorum queues for one of our deployments. So fare we did not have any issue with them. They also seem to survive restarts without issues (however reply queues are still broken afterwards in a small amount of cases, but they are no quorum/mirrored queues anyway).

So I would recommend them for everyone that creates a new cluster.

--

Felix Huettner

*From:* Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> *Sent:* Saturday, May 6, 2023 4:29 AM *To:* OpenStack Discuss <openstack-discuss@lists.openstack.org> *Subject:* [OPENSTACK][rabbitmq] using quorum queues

Hello guys.

IS there any guy who uses the quorum queue for openstack? Could you give some feedback to compare with classic queue?

Thank you.

Nguyen Huu Khoi

Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.

Hinweise zum Datenschutz finden Sie hier <https://www.datenschutz.schwarz>.

This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.

Information on data protection can be found here <https://www.datenschutz.schwarz>.

-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582

-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582

We are not using kolla on our side, sorry Le 18 juin 2023 04:04:39 GMT+02:00, Satish Patel <satish.txt@gmail.com> a écrit :

Great! This is good to know that Quorum is a good solution.

Do you have a config to enable in kolla-ansible deployment?

On Thu, Jun 15, 2023 at 5:43 AM Arnaud Morin <arnaud.morin@gmail.com> wrote:

...

Hey,

We are also using quorum in some regions and plan to enable quorum everwhere.

Note that we also manage to enable quorum for transient queues (using a custom patch as it's not doable with current oslo.messaging, see my request in [1]). We also introduced some custom changes in py-amqp to handle correctly the rabbit disconnections (see [2] and [3]).

So far, the real improvment is achieved thanks to the combination of all of these changes, enabling quorum queue only was not enough for us to notice any improvment.

The downside of quorum queues is that it consume more power on the rabbit cluster: you need more IO, CPU, RAM and network bandwith for the same number of queues (see [4]). It has to be taken into account.

Cheers, Arnaud.

[1] https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033343.ht... [2] https://github.com/celery/py-amqp/pull/410 [3] https://github.com/celery/py-amqp/pull/405 [4] https://plik.ovh/file/nHCny7psDCTrEm76/Pq9ASO9wUd8HRk4C/s_1686817000.png

On 13.06.23 - 09:14, Sa Pham wrote:

...

Dear Khôi,

Thanks for your reply.

On Tue, Jun 13, 2023 at 9:05 AM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:

...

Hello. Firstly, when I used the classic queue and sometimes, my rabbitmq cluster was broken, the computers showed state down and I needed to restart the computer service to make it up. Secondly, 1 of 3 controller is down but my system still works although it is not very first as fully controller. I ran it for about 3 months compared with classic. My openstack is Yoga and use Kolla-Ansible as a deployment tool, Nguyen Huu Khoi

On Tue, Jun 13, 2023 at 8:43 AM Sa Pham <saphi070@gmail.com> wrote:

...

Hi Khôi,

Why do you say using the quorum queue is more stable than the classic queue ?

Thanks,

On Tue, Jun 13, 2023 at 7:26 AM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:

...

Hello Huettner, I have used the quorum queue since March and it is ok until now. It looks more stable than the classic queue. Some feedback to you. Thank you. Nguyen Huu Khoi.

On Mon, May 8, 2023 at 1:14 PM Felix Hüttner <felix.huettner@mail.schwarz> wrote:

> Hi Nguyen, > > > > we are using quorum queues for one of our deployments. So fare we did > not have any issue with them. They also seem to survive restarts without > issues (however reply queues are still broken afterwards in a small amount > of cases, but they are no quorum/mirrored queues anyway). > > > > So I would recommend them for everyone that creates a new cluster. > > > > -- > > Felix Huettner > > > > *From:* Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> > *Sent:* Saturday, May 6, 2023 4:29 AM > *To:* OpenStack Discuss <openstack-discuss@lists.openstack.org> > *Subject:* [OPENSTACK][rabbitmq] using quorum queues > > > > Hello guys. > > IS there any guy who uses the quorum queue for openstack? Could you > give some feedback to compare with classic queue? > > Thank you. > > Nguyen Huu Khoi > > Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur > für die Verwertung durch den vorgesehenen Empfänger bestimmt. > Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den > Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. > > Hinweise zum Datenschutz finden Sie hier > <https://www.datenschutz.schwarz>. > > > This e-mail may contain confidential content and is intended only for > the specified recipient/s. > If you are not the intended recipient, please inform the sender > immediately and delete this e-mail. > > Information on data protection can be found here > <https://www.datenschutz.schwarz>. >

-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582

-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582

Thank you! I will deploy it with Quorum Queue and give you my feedback. On Sun, Jun 18, 2023 at 8:02 PM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:

Helo, With kolla ansible

nano /etc/kolla/config/global.conf

[oslo_messaging_rabbit] rabbit_quorum_queue = true

but you need destroy (rm rabbitmq container and its volume ) and redeploy new one to make quorum queues work.

Nguyen Huu Khoi

On Sun, Jun 18, 2023 at 9:04 AM Satish Patel <satish.txt@gmail.com> wrote:

...

Great! This is good to know that Quorum is a good solution.

Do you have a config to enable in kolla-ansible deployment?

On Thu, Jun 15, 2023 at 5:43 AM Arnaud Morin <arnaud.morin@gmail.com> wrote:

...

Hey,

We are also using quorum in some regions and plan to enable quorum everwhere.

Note that we also manage to enable quorum for transient queues (using a custom patch as it's not doable with current oslo.messaging, see my request in [1]). We also introduced some custom changes in py-amqp to handle correctly the rabbit disconnections (see [2] and [3]).

So far, the real improvment is achieved thanks to the combination of all of these changes, enabling quorum queue only was not enough for us to notice any improvment.

The downside of quorum queues is that it consume more power on the rabbit cluster: you need more IO, CPU, RAM and network bandwith for the same number of queues (see [4]). It has to be taken into account.

Cheers, Arnaud.

[1] https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033343.ht... [2] https://github.com/celery/py-amqp/pull/410 [3] https://github.com/celery/py-amqp/pull/405 [4] https://plik.ovh/file/nHCny7psDCTrEm76/Pq9ASO9wUd8HRk4C/s_1686817000.png

...

Dear Khôi,

Thanks for your reply.

On Tue, Jun 13, 2023 at 9:05 AM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:

...

Hello. Firstly, when I used the classic queue and sometimes, my rabbitmq cluster was broken, the computers showed state down and I needed to restart

On 13.06.23 - 09:14, Sa Pham wrote: the

...

...

computer service to make it up. Secondly, 1 of 3 controller is down but my system still works although it is not very first as fully controller. I ran it for about 3 months compared with classic. My openstack is Yoga and use Kolla-Ansible as a deployment tool, Nguyen Huu Khoi

On Tue, Jun 13, 2023 at 8:43 AM Sa Pham <saphi070@gmail.com> wrote:

...

Hi Khôi,

Why do you say using the quorum queue is more stable than the classic queue ?

Thanks,

On Tue, Jun 13, 2023 at 7:26 AM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:

> Hello Huettner, > I have used the quorum queue since March and it is ok until now. It > looks more stable than the classic queue. Some feedback to you. > Thank you. > Nguyen Huu Khoi. > > > > On Mon, May 8, 2023 at 1:14 PM Felix Hüttner <felix.huettner@mail.schwarz> > wrote: > >> Hi Nguyen, >> >> >> >> we are using quorum queues for one of our deployments. So fare we did >> not have any issue with them. They also seem to survive restarts without >> issues (however reply queues are still broken afterwards in a small amount >> of cases, but they are no quorum/mirrored queues anyway). >> >> >> >> So I would recommend them for everyone that creates a new cluster. >> >> >> >> -- >> >> Felix Huettner >> >> >> >> *From:* Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> >> *Sent:* Saturday, May 6, 2023 4:29 AM >> *To:* OpenStack Discuss <openstack-discuss@lists.openstack.org> >> *Subject:* [OPENSTACK][rabbitmq] using quorum queues >> >> >> >> Hello guys. >> >> IS there any guy who uses the quorum queue for openstack? Could you >> give some feedback to compare with classic queue? >> >> Thank you. >> >> Nguyen Huu Khoi >> >> Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur >> für die Verwertung durch den vorgesehenen Empfänger bestimmt. >> Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den >> Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. >> >> Hinweise zum Datenschutz finden Sie hier >> <https://www.datenschutz.schwarz>. >> >> >> This e-mail may contain confidential content and is intended only for >> the specified recipient/s. >> If you are not the intended recipient, please inform the sender >> immediately and delete this e-mail. >> >> Information on data protection can be found here >> <https://www.datenschutz.schwarz>. >> >

-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582

-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582

Hey, 1. I'm not sure if that's a proper community goal, simply because it boils down to individual deployments and affects a really minority of projects, like oslo.messaging and deployment projects. Moreover, this affects only rabbit driver, which is the most popular option in oslo.messaging but not the only one. Usually community goal is used when you need to apply changes to the absolute majority of services, like in case with SQLAlchemy. So here I think we should discuss more default behaviour of oslo.messaging for rabbit driver and default value of rabbit_quorum_queue, which is basically up to oslo team to decide. 2. In OpenStack-Ansible we already agreed on the latest PTG to make Quorum queues a new default and introduce migration to them. So it should be inlcuded in 2023.2 (bobcat) release, with keeping the upgrade path at very least to 2024.1 which will be the next SLURP. пт, 30 июн. 2023 г. в 11:05, Christian Rohmann <christian.rohmann@inovex.de>:

On 18/06/2023 04:04, Satish Patel wrote:

...

Great! This is good to know that Quorum is a good solution.

Everybody in this thread seems to have good experiences with using quorum queues and also RabbitMQ themselves clearly communicate that Quorum Queues are the future and classic queues will be gone soon:

* "Classic mirrored queues were deprecated in RabbitMQ version 3.9" - https://www.rabbitmq.com/migrate-mcq-to-qq.html * https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/#remov...

I wonder why there is not more effort to make this shift by ...

a) Why this is not an OpenStack tc goal for all services to use quorum queues by default initialize any new queues as quorum queues? Especially since "An oslo.messaging-compatible message queue" is one of the base services [2] overseen by the TC. RabbitMQ comes up time and again when talking about operational issues when running OpenStack. Steps to ensure this vital piece runs as smooth as possible is certainly worth discussing in my humble opinion.

I don't know if anyone on the TC reads this, but does it make sense to propose such a goal to https://opendev.org/openstack/governance ?

b) Have deployment tooling like openstack-ansible, kolla-ansible, ... support setting the oslo.messaging options [1] or even make quorum queues the new default

Regards

Christian

[1] https://docs.openstack.org/releasenotes/oslo.messaging/yoga.html#relnotes-12... [2] https://governance.openstack.org/tc/reference/base-services.html#current-lis...

...

2. In OpenStack-Ansible we already agreed on the latest PTG to make Quorum queues a new default and introduce migration to them. So it should be inlcuded in 2023.2 (bobcat) release, with keeping the upgrade path at very least to 2024.1 which will be the next SLURP.

Do you have a pointer to those efforts for me to follow? A Gerrit topic or bug?

Sure, you can follow this topic: https://review.opendev.org/q/topic:osa%252Fquorum_queues I haven't used <rabbit_transient_quorum_queue> as it's not merged yet in oslo (and I was unaware of this patch). So it sounds like it is worth waiting for it before proceeding with the topic.

[1] https://www.rabbitmq.com/migrate-mcq-to-qq.html#mcq-changes-way-queue-is-use... which [2] https://review.opendev.org/c/openstack/oslo.messaging/+/888479 [3] https://review.opendev.org/c/openstack/oslo.messaging/+/888479

PS: your [2] and [3] links are exactly the same.

Aha, ok, the same link I'm referring to in a patch [1] then :D [1] https://review.opendev.org/c/openstack/openstack-ansible/+/873618 пт, 14 июл. 2023 г. в 13:06, Christian Rohmann <christian.rohmann@inovex.de>:

On 14/07/2023 12:37, Dmitriy Rabotyagov wrote:

...

PS: your [2] and [3] links are exactly the same. Sorry. I really should have sent out the email after lunch ;-)

[2] was supposed to be https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/#remov... in regards to be saying:

...

With the deprecation of classic mirrored queues within RabbitMQ this migration needs to happen sooner or later anyways.

In short: before RabbitMQ 4.x the classic queues need to be gone for good. No more config option, no looking back.

Regards

Christian

Hi all! Fwiw here is a first Patchset to make the usage of quorum queues easily possible in kolla-ansible by providing a single on/off toggle: https://review.opendev.org/c/openstack/kolla-ansible/+/898543 It's still early and rather untested, so your feedback is appreciated. Thanks -- Sven Kieske Senior Cloud Engineer Mail: kieske@osism.tech Web: https://osism.tech OSISM GmbH Teckstraße 62 / 70190 Stuttgart / Deutschland Geschäftsführer: Christian Berendt Unternehmenssitz: Stuttgart Amtsgericht: Stuttgart, HRB 756139