Open Stack

On 13/04/2023 23:04, Satish Patel wrote:
> Thank Sean/Matt,
>
> This is interesting that I have only option to use classic mirror with durable :( because without mirror cluster acting up when one of node is down.

If you have planned maintenance and non-mirrored transient queues, you 
can first try draining the node to be removed, before removing it from 
the cluster. In my testing at least, this appears to be much more 
successful than relying on the RabbitMQ clients to do the failover and 
recreate queues.  See [1], or for RMQ <3.8 you can cobble something 
together with ha-mode nodes [2].

[1] https://www.rabbitmq.com/upgrade.html#maintenance-mode

[2] https://www.rabbitmq.com/ha.html#mirroring-arguments

This obviously doesn't solve the case of when a controller fails 
unexpectedly.

I also think it's worth making the distinction between a highly 
available messaging infrastructure, and queue mirroring. In many cases, 
if a RMQ node hosting a non-mirrored, transient queue goes down, it 
should be possible for a service to just recreate the queue on another 
node and retry. This often seems to fail, which leads to queue mirroring 
getting turned on.

>
> How people are scaling rabbitMQ for large scale?
>
> Sent from my iPhone
>
>> On Apr 13, 2023, at 7:50 AM, Sean Mooney <smooney at redhat.com> wrote:
>>
>> On Thu, 2023-04-13 at 09:07 +0100, Matt Crees wrote:
>>> Hi all, I'll reply in turn here:
>>>
>>> Radek, I agree it definitely will be a good addition to the KA docs.
>>> I've got it on my radar, will aim to get a patch proposed this week.
>>>
>>> Satish, I haven't personally been able to test durable queues on a
>>> system that large. According to the RabbitMQ docs
>>> (https://www.rabbitmq.com/queues.html#durability), "Throughput and
>>> latency of a queue is not affected by whether a queue is durable or
>>> not in most cases." However, I have anecdotally heard that it can
>>> affect some performance in particularly large systems.
>> the perfomacne of durable queue is dominated by the disk io
>> put rabbit on a pcie nvme ssd and it will have littel effect
>> use spinnign rust (a HDD even in raid 10) and the iops/throput of the
>> storage sued to make the qure durible (that just mean write all messages to disk)
>> will be the bottlenck for scaleablity
>> combine that with HA and that is worse as it has to be writen to multipel servers.
>> i belive that teh mirror implemantion will wait for all copies to be persisted
>> but i have not really looked into it. it was raised as a pain point with rabbit by
>> operator in the past in terms fo scaling.
>>
>>> Please note that if you are using the classic mirrored queues, you
>>> must also have them durable. Transient (i.e. non-durable) mirrored
>>> queues are not a supported feature and do cause bugs. (For example the
>>> "old incarnation" errors seen here:
>>> https://bugs.launchpad.net/kolla-ansible/+bug/1954925)
>>>
>>> Nguyen, I can confirm that we've seen the same behaviour. This is
>>> caused by a backported change in oslo.messaging (I believe you linked
>>> the relevant bug report previously). There is a fix in progress to fix
>>> this (https://review.opendev.org/c/openstack/oslo.messaging/+/866617),
>>> and in the meantime setting kombu_reconnect_delay < 1.0 does resolve
>>> it.
>>>
>>> Cheers,
>>> Matt
>>>
>>>> On Thu, 13 Apr 2023 at 03:04, Nguyễn Hữu Khôi <nguyenhuukhoinw at gmail.com> wrote:
>>>>
>>>> update:
>>>> I use SAN as Cinder backend.
>>>> Nguyen Huu Khoi
>>>>
>>>>
>>>> On Thu, Apr 13, 2023 at 9:02 AM Nguyễn Hữu Khôi <nguyenhuukhoinw at gmail.com> wrote:
>>>>> Hello guys.
>>>>>
>>>>> I do many tests on xena and yoga. then i am sure that without ha-queue and kombu_reconnect_delay=0.5(it can < 1)
>>>>> you cannot launch instances when 1 of 3 controllers is down.
>>>>> Somebody can verify what I say, I hope we will have a common solution for this problem because those who use openstack for the first time will continue to ask questions like that.
>>>>>
>>>>> Nguyen Huu Khoi
>>>>>
>>>>>
>>>>> On Thu, Apr 13, 2023 at 12:59 AM Satish Patel <satish.txt at gmail.com> wrote:
>>>>>> This is great! Matt,
>>>>>>
>>>>>> Documentation would be greatly appreciated. I have a counter question: does Durable queue be good for large clouds with 1000 compute nodes or better to not use durable queue. This is a private cloud and we don't care about persistent data.
>>>>>>
>>>>>> On Wed, Apr 12, 2023 at 12:37 PM Radosław Piliszek <radoslaw.piliszek at gmail.com> wrote:
>>>>>>> Hi Matt,
>>>>>>>
>>>>>>>> As you're now reconfiguring a running deployment, there are
>>>>>>>> some extra steps that need to be taken to migrate to durable queues.
>>>>>>>>
>>>>>>>> 1. You will need to stop the OpenStack services which use RabbitMQ.
>>>>>>>>
>>>>>>>> 2. Reset the state of RabbitMQ one each rabbit node with the following
>>>>>>>> commands. You must run each command on all RabbitMQ nodes before
>>>>>>>> moving on to the next command. This will remove all queues.
>>>>>>>>
>>>>>>>>     rabbitmqctl stop_app
>>>>>>>>     rabbitmqctl force_reset
>>>>>>>>     rabbitmqctl start_app
>>>>>>>>
>>>>>>>> 3. Start the OpenStack services again, at which point they will
>>>>>>>> recreate the appropriate queues as durable.
>>>>>>> This sounds like a great new addition-to-be to the Kolla Ansible docs!
>>>>>>> Could you please propose it as a change?
>>>>>>>
>>>>>>> Kindest,
>>>>>>> Radek
>>>>>>>