<div>                That fixed it; thank you! My co-workers will think I'm an RMQ guru.<br>            </div>            <div class="yahoo_quoted" style="margin:10px 0px 0px 0.8ex;border-left:1px solid #ccc;padding-left:1ex;">                        <div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">                                <div>                    On Monday, March 21, 2022, 10:52:16 AM EDT, Felix Hüttner <felix.huettner@mail.schwarz> wrote:                </div>                <div><br></div>                <div><br></div>                <div><div id="yiv7291928012"><style><!--#yiv7291928012   _filtered {} _filtered {} _filtered {}#yiv7291928012  #yiv7291928012 p.yiv7291928012MsoNormal, #yiv7291928012 li.yiv7291928012MsoNormal, #yiv7291928012 div.yiv7291928012MsoNormal   {margin:0cm;font-size:11.0pt;font-family:"Calibri", sans-serif;}#yiv7291928012 a:link, #yiv7291928012 span.yiv7291928012MsoHyperlink  {color:blue;text-decoration:underline;}#yiv7291928012 p.yiv7291928012msonormal, #yiv7291928012 li.yiv7291928012msonormal, #yiv7291928012 div.yiv7291928012msonormal     {margin-right:0cm;margin-left:0cm;font-size:11.0pt;font-family:"Calibri", sans-serif;}#yiv7291928012 .yiv7291928012MsoChpDefault      {font-family:"Calibri", sans-serif;} _filtered {}#yiv7291928012 div.yiv7291928012WordSection1 {}--></style><div><div class="yiv7291928012WordSection1"><p class="yiv7291928012MsoNormal"><span style="">Hi Albert,</span></p> <p class="yiv7291928012MsoNormal"><span style="">  </span></p> <p class="yiv7291928012MsoNormal"><span lang="EN-US" style="">i think the following should work (but not tested)</span></p> <p class="yiv7291928012MsoNormal"><span lang="EN-US" style="">  </span></p> <p class="yiv7291928012MsoNormal"><span lang="EN-US">{"vhost": "/", "name": "notifications-expire", "pattern": "^(notifications_designate|versioned_notifications).*", "apply-to": "queues", "definition": {"message-ttl":600000,"expires":1200000, "ha-mode":"all","ha-promote-on-shutdown": "always", "ha-sync-mode":"automatic"}, "priority":1},<br clear="none">{"vhost": "/", "name": "ha-all", "pattern": "^(?!(amq\.)|(.*_fanout_)|(reply_)).*", "apply-to": "all", "definition": {"ha-mode":"all","ha-promote-on-shutdown": "always", "ha-sync-mode":"automatic"}, "priority":0}</span></p> <p class="yiv7291928012MsoNormal"><span lang="EN-US">  </span></p> <p class="yiv7291928012MsoNormal"><span lang="EN-US">Note the “priority”: 1 on the first one. It should ensure it is prioritized over the second one.</span></p> <p class="yiv7291928012MsoNormal"><span lang="EN-US">This way the notifications get all the ha setting as well as the expire stuff.</span><span lang="EN-US" style=""></span></p> <p class="yiv7291928012MsoNormal"><span lang="EN-US" style="">  </span></p> <p class="yiv7291928012msonormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">Best Regards<br clear="none"><br clear="none">Felix Hüttner<br clear="none">Schwarz IT KG * Stiftsbergstraße 1 * D-74172 Neckarsulm</span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <div id="yiv7291928012yqt58994" class="yiv7291928012yqt2687661417"><div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm;"><p class="yiv7291928012MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Albert Braden <ozzzo@yahoo.com><br clear="none"><b>Sent:</b> Monday, March 21, 2022 2:56 PM<br clear="none"><b>To:</b> Openstack-discuss <openstack-discuss@lists.openstack.org>; Felix Hüttner <felix.huettner@mail.schwarz><br clear="none"><b>Subject:</b> Re: [kolla] [rabbitmq] RMQ trouble after config update</span></p> </div><p class="yiv7291928012MsoNormal"><span lang="EN-US">  </span></p> <div><p class="yiv7291928012MsoNormal">Thank you Felix; that’s very helpful. I think I understand how it works now. The problem is, I want the notifications_designate messages to expire, because if they don’t, unconsumed messages linger forever. Is it possible to combine these two policies so that all durable queues are HA, but notifications_designate still expires? Can I have two definitions applied to different patterns in a single policy?<br clear="none"><br clear="none">{"vhost": "/", "name": "notifications-expire", "pattern": "^(notifications_designate|versioned_notifications).*", "apply-to": "queues", "definition": {"message-ttl":600000,"expires":1200000}, "priority":0},<br clear="none">{"vhost": "/", "name": "ha-all", "pattern": "^(?!(amq\.)|(.*_fanout_)|(reply_)).*", "apply-to": "all", "definition": {"ha-mode":"all","ha-promote-on-shutdown": "always", "ha-sync-mode":"automatic"}, "priority":0} ] }</p> </div><div style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:7.5pt;"><div><div><p class="yiv7291928012MsoNormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">On Monday, March 21, 2022, 06:11:23 AM EDT, Felix Hüttner <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:felix.huettner@mail.schwarz" target="_blank" href="mailto:felix.huettner@mail.schwarz">felix.huettner@mail.schwarz</a>> wrote:</span></p> </div><div><p class="yiv7291928012MsoNormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">  </span></p> </div><div><p class="yiv7291928012MsoNormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">  </span></p> </div><div><div id="yiv7291928012"><div><div><p class="yiv7291928012msonormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">Hi Albert,</span></p> <p class="yiv7291928012msonormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"> </span></p> <p class="yiv7291928012msonormal"><span lang="EN-US" style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">TL;DR: you set durable on your openstack service, not on rabbitmq. It is the same for all queues, so either you make them all HA or none of them (at least afaik).</span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <p class="yiv7291928012msonormal"><span lang="EN-US" style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"> </span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <p class="yiv7291928012msonormal"><span lang="EN-US" style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">the „durable“ setting is not defined by the policy but by the client that declares the queue. Openstack normally uses the following setting for that:</span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <p class="yiv7291928012msonormal"><span lang="EN-US" style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"> </span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <p class="yiv7291928012msonormal"><span lang="EN-US" style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">[oslo_messaging_rabbit]</span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <p class="yiv7291928012msonormal"><span lang="EN-US" style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">amqp_durable_queues = true</span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <p class="yiv7291928012msonormal"><span lang="EN-US" style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"> </span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <p class="yiv7291928012msonormal"><span lang="EN-US" style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">AFAIK this is a global setting and not different for notifications vs. other queues.</span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <p class="yiv7291928012msonormal"><span lang="EN-US" style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">An exception of this is fanout/reply queues which are always non-durable.</span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <p class="yiv7291928012msonormal"><span lang="EN-US" style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"> </span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <p class="yiv7291928012msonormal"><span lang="EN-US" style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">The whole durable/HA thing can be summarized as follows</span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <p class="yiv7291928012msonormal"><span lang="EN-US" style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"> </span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <p class="yiv7291928012msonormal"><span lang="EN-US" style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">Durable: If true then the message will survive the crash/restart of a rabbitmq nodes. If false the message is lost on crash/restart. Set by the client declaring the queue</span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <p class="yiv7291928012msonormal"><span lang="EN-US" style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">HA-mode: If enabled the queue is replicated across multiple nodes. If disabled only a single node hosts the queue and all messages. Set by the policy on rabbitmq</span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <p class="yiv7291928012msonormal"><span lang="EN-US" style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"> </span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <p class="yiv7291928012msonormal"><span lang="EN-US" style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">And the following are the result of restarts of a single node in each of the combinations:</span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <table border="0" cellspacing="0" cellpadding="0" style="border-collapse:collapse;" class="yiv7291928012MsoNormalTable"><tbody><tr><td colspan="1" rowspan="1" valign="top" width="209" style="width:156.6pt;border:solid windowtext 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;"><p class="yiv7291928012msonormal"><span lang="EN-US"> </span></p> </td><td colspan="1" rowspan="1" valign="top" width="209" style="width:156.6pt;border:solid windowtext 1.0pt;border-left:none;padding:0cm 5.4pt 0cm 5.4pt;"><p class="yiv7291928012msonormal"><span lang="EN-US">Durable</span></p> </td><td colspan="1" rowspan="1" valign="top" width="209" style="width:156.6pt;border:solid windowtext 1.0pt;border-left:none;padding:0cm 5.4pt 0cm 5.4pt;"><p class="yiv7291928012msonormal"><span lang="EN-US">Non-durable</span></p> </td></tr><tr><td colspan="1" rowspan="1" valign="top" width="209" style="width:156.6pt;border:solid windowtext 1.0pt;border-top:none;padding:0cm 5.4pt 0cm 5.4pt;"><p class="yiv7291928012msonormal"><span lang="EN-US">HA</span></p> </td><td colspan="1" rowspan="1" valign="top" width="209" style="width:156.6pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;"><p class="yiv7291928012msonormal"><span lang="EN-US">Queue is failing over to another node. All messages are preserved</span></p> </td><td colspan="1" rowspan="1" valign="top" width="209" style="width:156.6pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;"><p class="yiv7291928012msonormal"><span lang="EN-US">DO NOT DO THIS. The queue should fail over to another node, but the other node does not have any message. This does not make sense and causes all kind of strange rabbitmq issues.</span></p> </td></tr><tr><td colspan="1" rowspan="1" valign="top" width="209" style="width:156.6pt;border:solid windowtext 1.0pt;border-top:none;padding:0cm 5.4pt 0cm 5.4pt;"><p class="yiv7291928012msonormal"><span lang="EN-US">Non-HA</span></p> </td><td colspan="1" rowspan="1" valign="top" width="209" style="width:156.6pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;"><p class="yiv7291928012msonormal"><span lang="EN-US">Queue is unavailable (no reads or writes) until the single node comes back. All messages are preserved.</span></p> </td><td colspan="1" rowspan="1" valign="top" width="209" style="width:156.6pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;"><p class="yiv7291928012msonormal"><span lang="EN-US">Queue is deleted and all messages are lost. It can be redeclared on another node while the first one is down.</span></p> </td></tr></tbody></table><p class="yiv7291928012msonormal"><span lang="EN-US" style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"> </span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <p class="yiv7291928012msonormal"><span lang="EN-US" style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"> </span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <p class="yiv7291928012msonormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">Best Regards<br clear="none"><br clear="none">Felix Hüttner<br clear="none">Schwarz IT KG * Stiftsbergstraße 1 * D-74172 Neckarsulm</span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <div id="yiv7291928012yqt24380"><div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm;"><p class="yiv7291928012msonormal"><b><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">From:</span></b><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"> Albert Braden <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:ozzzo@yahoo.com" target="_blank" href="mailto:ozzzo@yahoo.com">ozzzo@yahoo.com</a>><br clear="none"><b>Sent:</b> Friday, March 18, 2022 9:50 PM<br clear="none"><b>To:</b> Openstack-discuss <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:openstack-discuss@lists.openstack.org" target="_blank" href="mailto:openstack-discuss@lists.openstack.org">openstack-discuss@lists.openstack.org</a>>; Felix Hüttner <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:felix.huettner@mail.schwarz" target="_blank" href="mailto:felix.huettner@mail.schwarz">felix.huettner@mail.schwarz</a>><br clear="none"><b>Subject:</b> Re: [kolla] [rabbitmq] RMQ trouble after config update</span></p> </div><p class="yiv7291928012msonormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"> </span></p> <div><p class="yiv7291928012msonormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">I fixed the typo and redeployed. The filter now reads:<br clear="none"><br clear="none">{"vhost": "/", "name": "ha-all", "pattern": "^(?!(amq\.)|(.*_fanout_)|(reply_)|(notifications_designate)|(versioned_notifications)).*", "apply-to": "all", "definition": {"ha-mode":"all","ha-promote-on-shutdown": "always", "ha-sync-mode":"automatic"}, "priority":0} ]<br clear="none"><br clear="none">I also tried deleting the RMQ containers and volumes and then redeploying, and deleting the queues and their exchanges, but they are re-created durable.</span></p> </div><div style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:7.5pt;"><div><div><p class="yiv7291928012msonormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">On Friday, March 18, 2022, 04:22:17 PM EDT, Albert Braden <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:ozzzo@yahoo.com" target="_blank" href="mailto:ozzzo@yahoo.com">ozzzo@yahoo.com</a>> wrote:</span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> </div><div><p class="yiv7291928012msonormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"> </span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> </div><div><p class="yiv7291928012msonormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"> </span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> </div><div><div id="yiv7291928012"><div><div><p class="yiv7291928012msonormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">I did not intend to make the notifications_designate queues HA nor durable. It<br clear="none">appears that those two queues are treated differently from the others that I<br clear="none">didn't specify as HA. For example, if I look at one of the _fanout queues, it<br clear="none">is not durable:<br clear="none"><br clear="none">Features<br clear="none">x-expires: 1800000<br clear="none">Policy<br clear="none">Operator policy<br clear="none">Effective policy definition<br clear="none">Node rabbit@de6-ctrl2<br clear="none"><br clear="none">I tried adding the notifications_designate queues to the exclusion list:<br clear="none"><br clear="none">{"vhost": "/", "name": "ha-all", "pattern":<br clear="none">"^(?!(amq\.)|(.*_fanout_)|(reply_)(notifications_designate)|(versioned_notifications)).*",<br clear="none">"apply-to": "all", "definition": {"ha-mode":"all","ha-promote-on-shutdown":<br clear="none">"always", "ha-sync<br clear="none"><br clear="none">But this doesn't seem to make a difference. The notifications_designate queues<br clear="none">still have " Features: durable: true"<br clear="none"><br clear="none">How can I make the notifications_designate queues non-durable? </span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> </div><div style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:7.5pt;"><div><div id="yiv7291928012yqtfd37171"><div><p class="yiv7291928012msonormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">On Friday, March 18, 2022, 10:04:08 AM EDT, Felix Hüttner <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:felix.huettner@mail.schwarz" target="_blank" href="mailto:felix.huettner@mail.schwarz">felix.huettner@mail.schwarz</a>> wrote:</span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> </div><div><p class="yiv7291928012msonormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"> </span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> </div><div><p class="yiv7291928012msonormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"> </span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> </div><div><div><p class="yiv7291928012msonormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">Hi Albert,<br clear="none"><br clear="none">for the queue "notifications_designate.info" both the "notifications-expire" and the "ha-all" policy would match with their pattern. However only one policy can be applied to a queue at any given point in time. I assume "notifications-expire" is applied (but you can check that easily in the UI or with "rabbitmqctl list_queues name policy").<br clear="none"><br clear="none">In this case the "notifications_designate.info" is created as durable (because of amqp_durable_queues) and non-ha (as the policy does not define ha).<br clear="none">When a node with a durable and non-ha queue goes down then this queue no longer useable until either the node comes back or the queue is deleted.<br clear="none"><br clear="none">In your case I would assume that you also want this queue to be HA, so you probably need to set the options of the "ha-all" policy also in the "notifications-expire" policy.<br clear="none"><br clear="none">Regarding the last point of the queue being on ctrl2 and you connecting to ctrl1 this is normal rabbitmq behaviour in that it will forward messages to other nodes if needed.<br clear="none"><br clear="none">Best Regards<br clear="none"><br clear="none">Felix Hüttner<br clear="none">Schwarz IT KG * Stiftsbergstraße 1 * D-74172 Neckarsulm</span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <div id="yiv7291928012yqtfd61668"><p class="yiv7291928012msonormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"><br clear="none">-----Original Message-----<br clear="none">From: Albert Braden <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:ozzzo@yahoo.com" target="_blank" href="mailto:ozzzo@yahoo.com">ozzzo@yahoo.com</a>><br clear="none">Sent: Friday, March 18, 2022 1:54 PM<br clear="none">To: Openstack-discuss <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:openstack-discuss@lists.openstack.org" target="_blank" href="mailto:openstack-discuss@lists.openstack.org">openstack-discuss@lists.openstack.org</a>><br clear="none">Subject: [kolla] [rabbitmq] RMQ trouble after config update<br clear="none"><br clear="none">We're running kolla-ansible Train. I followed the recommendations in [1] and ended up with the following config:<br clear="none"><br clear="none">definitions.json (in the rabbitmq container):<br clear="none"><br clear="none">{<br clear="none">  "vhosts": [<br clear="none">    {"name": "/"}  ],<br clear="none">  "users": [<br clear="none">    {"name": "openstack", "password": "<password>", "tags": "administrator"}  ],<br clear="none">  "permissions": [<br clear="none">    {"user": "openstack", "vhost": "/", "configure": ".*", "write": ".*", "read": ".*"}  ],<br clear="none">  "policies":[<br clear="none">    {"vhost": "/", "name": "notifications-expire", "pattern": "^(notifications_designate|versioned_notifications).*", "apply-to": "queues", "definition": {"message-ttl":600000,"expires":1200000}, "priority":0},<br clear="none">    {"vhost": "/", "name": "ha-all", "pattern": "^(?!(amq\.)|(.*_fanout_)|(reply_)).*", "apply-to": "all", "definition": {"ha-mode":"all","ha-promote-on-shutdown": "always", "ha-sync-mode":"automatic"}, "priority":0}  ] }<br clear="none"><br clear="none">etc/kolla/config/global.conf:<br clear="none"><br clear="none">[oslo_messaging_rabbit]<br clear="none">amqp_durable_queues = True<br clear="none"><br clear="none">This fixed some issues, but we seem to have a new issue so I must be missing a setting. When we stop the RMQ container on ctrl1, designate stops working (DNS records are not created nor deleted) and I see this in designate-sink.log:<br clear="none"><br clear="none">2022-03-17 19:21:29.261 28 ERROR oslo.messaging._drivers.impl_rabbit [req-2c0cd9f4-5331-4697-9c3e-eece475a52af - - - - -] Failed to consume message from queue: Queue.declare: (404) NOT_FOUND - home node '<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:rabbit@de6-ctrl1" target="_blank" href="mailto:rabbit@de6-ctrl1">rabbit@de6-ctrl1</a>' of durable queue 'notifications_designate.info' in vhost '/' is down or inaccessible: amqp.exceptions.NotFound: Queue.declare: (404) NOT_FOUND - home node '<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:rabbit@de6-ctrl1" target="_blank" href="mailto:rabbit@de6-ctrl1">rabbit@de6-ctrl1</a>' of durable queue 'notifications_designate.info' in vhost '/' is down or inaccessible<br clear="none"><br clear="none">When I look at the notifications_designate.info queue in the web interface, it appears to have moved to ctrl2:<br clear="none"><br clear="none">Features<br clear="none">durable:              true<br clear="none">Policy    notifications-expire<br clear="none">Operator policy<br clear="none">Effective policy definition<br clear="none">expires: 1200000<br clear="none">message-ttl:      600000<br clear="none">Node    <a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:rabbit@qde3-ctrl2" target="_blank" href="mailto:rabbit@qde3-ctrl2">rabbit@qde3-ctrl2</a><br clear="none"><br clear="none">When I look at designate.conf in the designate_sink containers I don't see anything pointing only to ctrl1:<br clear="none"><br clear="none">transport_url = rabbit://openstack:<pwd>@<ctrl1>:5672,openstack:<pwd>@<ctrl2>:5672,openstack:<pwd>@<ctrl3>:5672//<br clear="none"><br clear="none">But it appears that Designate still tries to use the queue on ctrl1. After I bring up ctrl1, the notifications_designate.info queue remains on ctrl2, but Designate starts working.<br clear="none"><br clear="none">What am I missing?<br clear="none"><br clear="none">[1] </span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> </div><p class="yiv7291928012msonormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"><a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.openstack.org%2Fwiki%2FLarge_Scale_Configuration_Rabbit&data=04%7C01%7C%7C32d09b881bb2459c89d008da0b428c8c%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C637834677702617860%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=QTI4mDjems3GOf6pEOkPAlTqNA68SYvlUVmNQ%2BnVu%2Bg%3D&reserved=0">https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.openstack.org%2Fwiki%2FLarge_Scale_Configuration_Rabbit&data=04%7C01%7C%7C103b79868a61424923bb08da08df82f7%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C637832053312648154%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=1RIlfljNyKg71IrZc0NIuqDoHPkTGtSBthadz%2Bneofk%3D&reserved=0</a><br clear="none"><br clear="none">Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hier<<a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.datenschutz.schwarz%2F&data=04%7C01%7C%7C32d09b881bb2459c89d008da0b428c8c%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C637834677702617860%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=jlA5EviINuxe%2BjZU15IplSM3eWyaKco0OotB1PLdx3E%3D&reserved=0">https://www.datenschutz.schwarz</a></span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> <div id="yiv7291928012yqtfd33885"><p style="margin-bottom:12.0pt;" class="yiv7291928012msonormal"><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;">>.</span><span style="font-size:10.0pt;font-family:sans-serif;color:#26282A;"></span></p> </div></div></div></div></div></div></div></div></div></div></div></div></div><p><span style="color:#26282A;">Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie<a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.datenschutz.schwarz%2F&data=04%7C01%7C%7C32d09b881bb2459c89d008da0b428c8c%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C637834677702617860%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=jlA5EviINuxe%2BjZU15IplSM3eWyaKco0OotB1PLdx3E%3D&reserved=0">hier</a>.</span></p> </div></div></div></div></div></div></div><p style="font-family:Calibri;font-size:11pt;">Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie<a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://www.datenschutz.schwarz">hier</a>.</p></div></div></div>            </div>                </div>