Open Stack

Hello fellow OpenStackers,

you might be interested in what we (the Kolla team) have done to RMQ
policies: https://review.opendev.org/c/openstack/kolla-ansible/+/824994

-yoctozepto

On Fri, 18 Mar 2022 at 15:16, Albert Braden <ozzzo at yahoo.com> wrote:
>
> That's what I was missing; I'll experiment with this today. Thank you!
> On Friday, March 18, 2022, 10:04:08 AM EDT, Felix Hüttner <felix.huettner at mail.schwarz> wrote:
>
>
> Hi Albert,
>
> for the queue "notifications_designate.info" both the "notifications-expire" and the "ha-all" policy would match with their pattern. However only one policy can be applied to a queue at any given point in time. I assume "notifications-expire" is applied (but you can check that easily in the UI or with "rabbitmqctl list_queues name policy").
>
> In this case the "notifications_designate.info" is created as durable (because of amqp_durable_queues) and non-ha (as the policy does not define ha).
> When a node with a durable and non-ha queue goes down then this queue no longer useable until either the node comes back or the queue is deleted.
>
> In your case I would assume that you also want this queue to be HA, so you probably need to set the options of the "ha-all" policy also in the "notifications-expire" policy.
>
> Regarding the last point of the queue being on ctrl2 and you connecting to ctrl1 this is normal rabbitmq behaviour in that it will forward messages to other nodes if needed.
>
> Best Regards
>
> Felix Hüttner
> Schwarz IT KG * Stiftsbergstraße 1 * D-74172 Neckarsulm
>
> -----Original Message-----
> From: Albert Braden <ozzzo at yahoo.com>
> Sent: Friday, March 18, 2022 1:54 PM
> To: Openstack-discuss <openstack-discuss at lists.openstack.org>
> Subject: [kolla] [rabbitmq] RMQ trouble after config update
>
> We're running kolla-ansible Train. I followed the recommendations in [1] and ended up with the following config:
>
> definitions.json (in the rabbitmq container):
>
> {
>   "vhosts": [
>     {"name": "/"}  ],
>   "users": [
>     {"name": "openstack", "password": "<password>", "tags": "administrator"}  ],
>   "permissions": [
>     {"user": "openstack", "vhost": "/", "configure": ".*", "write": ".*", "read": ".*"}  ],
>   "policies":[
>     {"vhost": "/", "name": "notifications-expire", "pattern": "^(notifications_designate|versioned_notifications).*", "apply-to": "queues", "definition": {"message-ttl":600000,"expires":1200000}, "priority":0},
>     {"vhost": "/", "name": "ha-all", "pattern": "^(?!(amq\.)|(.*_fanout_)|(reply_)).*", "apply-to": "all", "definition": {"ha-mode":"all","ha-promote-on-shutdown": "always", "ha-sync-mode":"automatic"}, "priority":0}  ] }
>
> etc/kolla/config/global.conf:
>
> [oslo_messaging_rabbit]
> amqp_durable_queues = True
>
> This fixed some issues, but we seem to have a new issue so I must be missing a setting. When we stop the RMQ container on ctrl1, designate stops working (DNS records are not created nor deleted) and I see this in designate-sink.log:
>
> 2022-03-17 19:21:29.261 28 ERROR oslo.messaging._drivers.impl_rabbit [req-2c0cd9f4-5331-4697-9c3e-eece475a52af - - - - -] Failed to consume message from queue: Queue.declare: (404) NOT_FOUND - home node 'rabbit at de6-ctrl1' of durable queue 'notifications_designate.info' in vhost '/' is down or inaccessible: amqp.exceptions.NotFound: Queue.declare: (404) NOT_FOUND - home node 'rabbit at de6-ctrl1' of durable queue 'notifications_designate.info' in vhost '/' is down or inaccessible
>
> When I look at the notifications_designate.info queue in the web interface, it appears to have moved to ctrl2:
>
> Features
> durable:              true
> Policy    notifications-expire
> Operator policy
> Effective policy definition
> expires: 1200000
> message-ttl:      600000
> Node    rabbit at qde3-ctrl2
>
> When I look at designate.conf in the designate_sink containers I don't see anything pointing only to ctrl1:
>
> transport_url = rabbit://openstack:<pwd>@<ctrl1>:5672,openstack:<pwd>@<ctrl2>:5672,openstack:<pwd>@<ctrl3>:5672//
>
> But it appears that Designate still tries to use the queue on ctrl1. After I bring up ctrl1, the notifications_designate.info queue remains on ctrl2, but Designate starts working.
>
> What am I missing?
>
> [1]
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.openstack.org%2Fwiki%2FLarge_Scale_Configuration_Rabbit&data=04%7C01%7C%7C103b79868a61424923bb08da08df82f7%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C637832053312648154%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=1RIlfljNyKg71IrZc0NIuqDoHPkTGtSBthadz%2Bneofk%3D&reserved=0
>
> Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarz
> >.
>