<div>                Now that the holidays are over I'm trying this one again. Can anyone help me figure out how to set "expires" and "message-ttl" ?<br>            </div>            <div class="yahoo_quoted" style="margin:10px 0px 0px 0.8ex;border-left:1px solid #ccc;padding-left:1ex;">                        <div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">                                <div>                    On Thursday, December 16, 2021, 01:43:57 PM EST, Albert Braden <ozzzo@yahoo.com> wrote:                </div>                <div><br></div>                <div><br></div>                <div><div id="yiv0003717239"><div><div>                I tried these policies in ansible/roles/rabbitmq/templates/definitions.json.j2:<br clear="none"><br clear="none">  "policies":[<br clear="none">    {"vhost": "/", "name": "ha-all", "pattern": '^(?!(amq\.)|(.*_fanout_)|(reply_)).*', "apply-to": "all", "definition": {"ha-mode":"all"}, "priority":0}{% if project_name == 'outward_rabbitmq' %},<br clear="none">    {"vhost": "/", "name": "notifications-ttl", "pattern": "^(notifications|versioned_notifications)\\.", "apply-to": "queues", "definition": {"message-ttl":600}, "priority":0}<br clear="none">    {"vhost": "/", "name": "notifications-expire", "pattern": "^(notifications|versioned_notifications)\\.", "apply-to": "queues", "definition": {"expire":3600}, "priority":0}<br clear="none">    {"vhost": "{{ murano_agent_rabbitmq_vhost }}", "name": "ha-all", "pattern": ".*", "apply-to": "all", "definition": {"ha-mode":"all"}, "priority":0}<br clear="none">    {% endif %}<br clear="none"><br clear="none">But I still see unconsumed messages lingering in notifications_extractor.info. From reading the docs I think this setting should cause messages to expire after 600 seconds, and unused queues to be deleted after 3600 seconds. What am I missing?            </div>            <div id="yiv0003717239yqt05816" class="yiv0003717239yqt2346696467"><div style="margin:10px 0px 0px 0.8ex;border-left:1px solid #ccc;padding-left:1ex;" class="yiv0003717239yahoo_quoted">                        <div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">                                <div>                    On Tuesday, December 14, 2021, 04:18:09 PM EST, Albert Braden <ozzzo@yahoo.com> wrote:                </div>                <div><br clear="none"></div>                <div><br clear="none"></div>                <div><div id="yiv0003717239"><div><div>                Following [1] I successfully set "amqp_durable_queues = True" and restricted HA to the appropriate queues, but I'm having trouble with some of the other settings such as "expires" and "message-ttl". Does anyone have an example of a working kolla config that includes these changes?<br clear="none"><br clear="none">[1] https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit<br clear="none">            </div>            <div id="yiv0003717239yqt46676" class="yiv0003717239yqt7610968821"><div style="margin:10px 0px 0px 0.8ex;border-left:1px solid #ccc;padding-left:1ex;" class="yiv0003717239yahoo_quoted">                        <div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">                                <div>                    On Monday, December 13, 2021, 07:51:32 AM EST, Herve Beraud <hberaud@redhat.com> wrote:                </div>                <div><br clear="none"></div>                <div><br clear="none"></div>                <div><div id="yiv0003717239"><div><div dir="ltr"><div dir="ltr">So, your config snippet LGTM.<br clear="none"></div><br clear="none"><div id="yiv0003717239yqt51608" class="yiv0003717239yqt5810937397"><div class="yiv0003717239gmail_quote"><div dir="ltr" class="yiv0003717239gmail_attr">Le ven. 10 déc. 2021 à 17:50, Albert Braden <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:ozzzo@yahoo.com" target="_blank" href="mailto:ozzzo@yahoo.com">ozzzo@yahoo.com</a>> a écrit :<br clear="none"></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;" class="yiv0003717239gmail_quote"><div>                Sorry, that was a transcription error. I thought "True" and my fingers typed "False." The correct lines are:<br clear="none"><br clear="none">[oslo_messaging_rabbit]<br clear="none">amqp_durable_queues = True<br clear="none"><br clear="none">            </div>            <div style="margin:10px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;">                        <div style="font-family:Helvetica, Arial, sans-serif;font-size:13px;color:rgb(38,40,42);">                                <div>                    On Friday, December 10, 2021, 02:55:55 AM EST, Herve Beraud <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:hberaud@redhat.com" target="_blank" href="mailto:hberaud@redhat.com">hberaud@redhat.com</a>> wrote:                </div>                <div><br clear="none"></div>                <div><br clear="none"></div>                <div><div id="yiv0003717239gmail-m_2842749581596187191yiv1924437727"><div><div dir="ltr"><div>If you plan to let `amqp_durable_queues = False` (i.e if you plan to keep this config equal to false), then you don't need to add these config lines as this is already the default value [1].<br clear="none"></div><div><br clear="none"></div><div>[1] <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messaging/_drivers/amqp.py#L34">https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messaging/_drivers/amqp.py#L34</a></div></div><br clear="none"><div id="yiv0003717239gmail-m_2842749581596187191yiv1924437727yqt43679"><div><div dir="ltr">Le jeu. 9 déc. 2021 à 22:40, Albert Braden <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:ozzzo@yahoo.com" target="_blank" href="mailto:ozzzo@yahoo.com">ozzzo@yahoo.com</a>> a écrit :<br clear="none"></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;"><div>                Replying from my home email because I've been asked to not email the list from my work email anymore, until I get permission from upper management.<br clear="none"><br clear="none">I'm not sure I follow. I was planning to add 2 lines to etc/kolla/config/global.conf:<br clear="none"><br clear="none">[oslo_messaging_rabbit]<br clear="none">amqp_durable_queues = False<br clear="none"><br clear="none">Is that not sufficient? What is involved in configuring  dedicated control exchanges for each service? What would that look like in the config?<br clear="none"><br clear="none"><br clear="none">From: Herve Beraud <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:hberaud@redhat.com" target="_blank" href="mailto:hberaud@redhat.com">hberaud@redhat.com</a>> <br clear="none">Sent: Thursday, December 9, 2021 2:45 AM<br clear="none">To: Bogdan Dobrelya <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:bdobreli@redhat.com" target="_blank" href="mailto:bdobreli@redhat.com">bdobreli@redhat.com</a>><br clear="none">Cc: <a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:openstack-discuss@lists.openstack.org" target="_blank" href="mailto:openstack-discuss@lists.openstack.org">openstack-discuss@lists.openstack.org</a><br clear="none">Subject: [EXTERNAL] Re: [ops] [kolla] RabbitMQ High Availability<br clear="none"><br clear="none"> <br clear="none"><br clear="none">Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. <br clear="none"><br clear="none"> <br clear="none"><br clear="none"> <br clear="none"><br clear="none">Le mer. 8 déc. 2021 à 11:48, Bogdan Dobrelya <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:bdobreli@redhat.com" target="_blank" href="mailto:bdobreli@redhat.com">bdobreli@redhat.com</a>> a écrit :<br clear="none"><br clear="none">Please see inline<br clear="none"><br clear="none">>> I read this with great interest because we are seeing this issue. Questions:<br clear="none">>> <br clear="none">>> 1. We are running kola-ansible Train, and our RMQ version is 3.7.23. Should we be upgrading our Train clusters to use 3.8.x?<br clear="none">>> 2. Document [2] recommends policy '^(?!(amq\.)|(.*_fanout_)|(reply_)).*'. I don't see this in our ansible playbooks, nor in any of the config files in the RMQ container. What would this look like in Ansible, and what should the resulting container config look like?<br clear="none">>> 3. It appears that we are not setting "amqp_durable_queues = True". What does this setting look like in Ansible, and what file does it go into?<br clear="none">> <br clear="none">> Note that even having rabbit HA policies adjusted like that and its HA<br clear="none">> replication factor [0] decreased (e.g. to a 2), there still might be<br clear="none">> high churn caused by a large enough number of replicated durable RPC<br clear="none">> topic queues. And that might cripple the cloud down with the incurred<br clear="none">> I/O overhead because a durable queue requires all messages in it to be<br clear="none">> persisted to a disk (for all the messaging cluster replicas) before they<br clear="none">> are ack'ed by the broker.<br clear="none">> <br clear="none">> Given that said, Oslo messaging would likely require a more granular<br clear="none">> control for topic exchanges and the durable queues flag - to tell it to<br clear="none">> declare as durable only the most critical paths of a service. A single<br clear="none">> config setting and a single control exchange per a service might be not<br clear="none">> enough.<br clear="none"><br clear="none">Also note that therefore, amqp_durable_queue=True requires dedicated<br clear="none">control exchanges configured for each service. Those that use<br clear="none">'openstack' as a default cannot turn the feature ON. Changing it to a<br clear="none">service specific might also cause upgrade impact, as described in the<br clear="none">topic [3].<br clear="none"><br clear="none"> <br clear="none"><br clear="none">The same is true for `amqp_auto_delete=True`. That requires dedicated control exchanges else it won't work if each service defines its own policy on a shared control exchange (e.g `openstack`) and if policies differ from each other.<br clear="none"><br clear="none"> <br clear="none"><br clear="none">[3] <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://review.opendev.org/q/topic:scope-config-opts">https://review.opendev.org/q/topic:scope-config-opts</a><br clear="none"><br clear="none">> <br clear="none">> There are also race conditions with durable queues enabled, like [1]. A<br clear="none">> solution could be where each service declare its own dedicated control<br clear="none">> exchange with its own configuration.<br clear="none">> <br clear="none">> Finally, openstack components should add perhaps a *.next CI job to test<br clear="none">> it with durable queues, like [2]<br clear="none">> <br clear="none">> [0] <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://www.rabbitmq.com/ha.html#replication-factor">https://www.rabbitmq.com/ha.html#replication-factor</a><br clear="none">> <br clear="none">> [1]<br clear="none">> <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://zuul.opendev.org/t/openstack/build/aa514dd788f34cc1be3800e6d7dba0e8/log/controller/logs/screen-n-cpu.txt">https://zuul.opendev.org/t/openstack/build/aa514dd788f34cc1be3800e6d7dba0e8/log/controller/logs/screen-n-cpu.txt</a><br clear="none">> <br clear="none">> [2] <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://review.opendev.org/c/openstack/nova/+/820523">https://review.opendev.org/c/openstack/nova/+/820523</a><br clear="none">> <br clear="none">>> <br clear="none">>> Does anyone have a sample set of RMQ config files that they can share?<br clear="none">>> <br clear="none">>> It looks like my Outlook has ruined the link; reposting:<br clear="none">>> [2] <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit">https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit</a><br clear="none">> <br clear="none">> <br clear="none">> -- <br clear="none">> Best regards,<br clear="none">> Bogdan Dobrelya,<br clear="none">> Irc #bogdando<br clear="none"><br clear="none"><br clear="none">-- <br clear="none">Best regards,<br clear="none">Bogdan Dobrelya,<br clear="none">Irc #bogdando<br clear="none"><br clear="none"><br clear="none"><br clear="none"><br clear="none">--<br clear="none"><br clear="none">Hervé Beraud<br clear="none"><br clear="none">Senior Software Engineer at Red Hat<br clear="none"><br clear="none">irc: hberaud<br clear="none"><br clear="none"><a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://github.com/4383/">https://github.com/4383/</a><br clear="none"><br clear="none"><a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://twitter.com/4383hberaud">https://twitter.com/4383hberaud</a>            </div>            </blockquote></div></div><br clear="all"><br clear="none">-- <br clear="none"><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Hervé Beraud</div><div>Senior Software Engineer at Red Hat</div><div>irc: hberaud</div><div><a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://github.com/4383/">https://github.com/4383/</a></div><div><a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://twitter.com/4383hberaud">https://twitter.com/4383hberaud</a><br clear="none"><br clear="none"></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>            </div>                </div></blockquote></div></div><br clear="all"><br clear="none">-- <br clear="none"><div dir="ltr" class="yiv0003717239gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Hervé Beraud</div><div>Senior Software Engineer at Red Hat</div><div>irc: hberaud</div><div><a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://github.com/4383/">https://github.com/4383/</a></div><div><a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://twitter.com/4383hberaud">https://twitter.com/4383hberaud</a><br clear="none"><br clear="none"></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>            </div>                </div></div></div></div></div>            </div>                </div></div></div></div></div>            </div>                </div>