<div>                After reading more I realize that "expires" is also set in ms. So it looks like the correct settings are:<br><br>message-ttl: 60000<br>expires: 120000<br><br>This would expire messages in 10 minutes and queues in 20 minutes.<br><br>The only remaining question is, how can I specify these in a variable without generating the "not a valid message TTL" error?<br>            </div>            <div class="yahoo_quoted" style="margin:10px 0px 0px 0.8ex;border-left:1px solid #ccc;padding-left:1ex;">                        <div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">                                <div>                    On Thursday, January 13, 2022, 01:22:33 PM EST, Albert Braden <ozzzo@yahoo.com> wrote:                </div>                <div><br></div>                <div><br></div>                <div><div id="yiv5629114850"><div><div>                Update: I googled around and found this: https://tickets.puppetlabs.com/browse/MODULES-2986<br clear="none"><br clear="none">Apparently the " | int " isn't working. I tried '60000' and "60000" but that didn't make a difference. In desperation I tried this:<br clear="none"><br clear="none"> {"vhost": "/", "name": "notifications-expire", "pattern": "^(notifications|versioned_notifications).*", "apply-to": "queues", "definition": {"message-ttl":60000,"expires":1200}, "priority":0},<br clear="none"><br clear="none"> That works, but I'd prefer to use a variable. Has anyone done this successfully? Also, am I understanding correctly that "message-ttl" is set in milliseconds and "expires" is set in seconds? Or do I need to use ms for "expires" too?<br clear="none">            </div>            <div id="yiv5629114850yqt43470" class="yiv5629114850yqt0161403792"><div style="margin:10px 0px 0px 0.8ex;border-left:1px solid #ccc;padding-left:1ex;" class="yiv5629114850yahoo_quoted">                        <div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">                                <div>                    On Thursday, January 13, 2022, 11:03:11 AM EST, Albert Braden <ozzzo@yahoo.com> wrote:                </div>                <div><br clear="none"></div>                <div><br clear="none"></div>                <div><div id="yiv5629114850"><div><div>                After digging further I realized that I'm not setting TTL; only queue expiration. Here's what I see in the GUI when I look at affected queues:<br clear="none"><br clear="none">Policy    notifications-expire<br clear="none">Effective policy definition             expires: 1200<br clear="none"><br clear="none">This is what I have in definitions.json.j2:<br clear="none"><br clear="none">{"vhost": "/", "name": "notifications-expire", "pattern": "^(notifications|versioned_notifications).*", "apply-to": "queues", "definition": {"expires":1200}, "priority":0},<br clear="none"><br clear="none">I tried this to set both:<br clear="none"><br clear="none">{"vhost": "/", "name": "notifications-expire", "pattern": "^(notifications|versioned_notifications).*", "apply-to": "queues", "definition": {"message-ttl":"{{ rabbitmq_message_ttl | int }}","expires":1200}, "priority":0},<br clear="none"><br clear="none">But the RMQ containers restart every 60 seconds and puke this into the log:<br clear="none"><br clear="none">[error] <0.322.0> CRASH REPORT Process <0.322.0> with 0 neighbours exited with reason: {error,<<"<<\"Validation failed\\n\\n&lt;&lt;\\\"600\\\"&gt;&gt; is not a valid message TTL\\n (//notifications-expire)\">>">>} in application_master:init/4 line 138<br clear="none"><br clear="none">After reading the doc on TTL: https://www.rabbitmq.com/ttl.html I realized that the TTL is set in ms, so I tried "rabbitmq_message_ttl: 60000"<br clear="none"><br clear="none">but that only changes the number in the error:<br clear="none"><br clear="none">[error] <0.318.0> CRASH REPORT Process <0.318.0> with 0 neighbours exited with reason: {error,<<"<<\"Validation failed\\n\\n&lt;&lt;\\\"60000\\\"&gt;&gt; is not a valid message TTL\\n (//notifications-expire)\">>">>} in application_master:init/4 line 138<br clear="none"><br clear="none">What am I missing?<br clear="none"><br clear="none"><br clear="none">            </div>            <div id="yiv5629114850yqt74086" class="yiv5629114850yqt6465618837"><div style="margin:10px 0px 0px 0.8ex;border-left:1px solid #ccc;padding-left:1ex;" class="yiv5629114850yahoo_quoted">                        <div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">                                <div>                    On Wednesday, January 12, 2022, 05:11:41 PM EST, Dale Smith <dale@catalystcloud.nz> wrote:                </div>                <div><br clear="none"></div>                <div><br clear="none"></div>                <div><div id="yiv5629114850"><div>    <p>In the web interface(RabbitMQ 3.8.23, not using Kolla), when      looking at the queue you will see the "Policy" listed by name, and      "Effective policy definition".</p>    <p>You can either view the policy definition, and the arguments for      the definitions applied, or "effective policy definition" should      show you the list.<br clear="none">    </p>    <p><br clear="none">    </p>    <p>It may be relevant to note: "Each exchange or queue will have at      most one policy matching" -      <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://www.rabbitmq.com/parameters.html#how-policies-work" class="yiv5629114850moz-txt-link-freetext">https://www.rabbitmq.com/parameters.html#how-policies-work</a><br clear="none">    </p>    <p>I've added a similar comment to the linked patchset.<br clear="none">    </p>    <p><br clear="none">    </p>    <div id="yiv5629114850yqt08677" class="yiv5629114850yqt7615999720"><div class="yiv5629114850moz-cite-prefix">On 13/01/22 7:26 am, Albert Braden      wrote:<br clear="none">    </div>    <blockquote type="cite">      </blockquote></div></div><div id="yiv5629114850yqt47862" class="yiv5629114850yqt7615999720"><div><div> This is very helpful. Thank you! It appears that I have        successfully set the expire time to 1200, because I no longer        see unconsumed messages lingering in my queues, but it's not        obvious how to verify. In the web interface, when I look at the        queues, I see things like policy, state, features and consumers,        but I don't see a timeout or expire value, nor do I find the        number 1200 anywhere. Where should I be looking in the web        interface to verify that I set the expire time correctly? Or do        I need to use the CLI?<br clear="none">      </div>      <div style="margin:10px 0px 0px 0.8ex;border-left:1px solid #ccc;padding-left:1ex;" class="yiv5629114850yahoo_quoted">        <div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">          <div> On Wednesday, January 5, 2022, 04:23:29 AM EST, Mark            Goddard <a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:mark@stackhpc.com" target="_blank" href="mailto:mark@stackhpc.com" class="yiv5629114850moz-txt-link-rfc2396E"><mark@stackhpc.com></a> wrote: </div>          <div><br clear="none">          </div>          <div><br clear="none">          </div>          <div>            <div dir="ltr">On Tue, 4 Jan 2022 at 14:08, Albert Braden              <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:ozzzo@yahoo.com" target="_blank" href="mailto:ozzzo@yahoo.com">ozzzo@yahoo.com</a>>              wrote:<br clear="none">              ><br clear="none">              > Now that the holidays are over I'm trying this one              again. Can anyone help me figure out how to set "expires"              and "message-ttl" ?<br clear="none">              <br clear="none">              John Garbutt proposed a few patches for RabbitMQ in kolla,              including<br clear="none">              this: <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://review.opendev.org/c/openstack/kolla-ansible/+/822191">https://review.opendev.org/c/openstack/kolla-ansible/+/822191</a><br clear="none">              <br clear="none">              <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://review.opendev.org/q/hashtag:%2522rabbitmq%2522+">https://review.opendev.org/q/hashtag:%2522rabbitmq%2522+</a>(status:open+OR+status:merged)+project:openstack/kolla-ansible<br clear="none">              <br clear="none">              Note that they are currently untested.<br clear="none">              <br clear="none">              Mark              <div id="yiv5629114850yqtfd18026" class="yiv5629114850yqt2589214125"><br clear="none">                <br clear="none">                > On Thursday, December 16, 2021, 01:43:57 PM EST,                Albert Braden <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:ozzzo@yahoo.com" target="_blank" href="mailto:ozzzo@yahoo.com">ozzzo@yahoo.com</a>>                wrote:<br clear="none">                ><br clear="none">                ><br clear="none">                > I tried these policies in                ansible/roles/rabbitmq/templates/definitions.json.j2:<br clear="none">                ><br clear="none">                > "policies":[<br clear="none">                > {"vhost": "/", "name": "ha-all", "pattern":                '^(?!(amq\.)|(.*_fanout_)|(reply_)).*', "apply-to":                "all", "definition": {"ha-mode":"all"}, "priority":0}{%                if project_name == 'outward_rabbitmq' %},<br clear="none">                > {"vhost": "/", "name": "notifications-ttl",                "pattern":                "^(notifications|versioned_notifications)\\.",                "apply-to": "queues", "definition": {"message-ttl":600},                "priority":0}<br clear="none">                > {"vhost": "/", "name": "notifications-expire",                "pattern":                "^(notifications|versioned_notifications)\\.",                "apply-to": "queues", "definition": {"expire":3600},                "priority":0}<br clear="none">                > {"vhost": "{{ murano_agent_rabbitmq_vhost }}",                "name": "ha-all", "pattern": ".*", "apply-to": "all",                "definition": {"ha-mode":"all"}, "priority":0}<br clear="none">                > {% endif %}<br clear="none">                ><br clear="none">                > But I still see unconsumed messages lingering in                notifications_extractor.info. From reading the docs I                think this setting should cause messages to expire after                600 seconds, and unused queues to be deleted after 3600                seconds. What am I missing?<br clear="none">                > On Tuesday, December 14, 2021, 04:18:09 PM EST,                Albert Braden <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:ozzzo@yahoo.com" target="_blank" href="mailto:ozzzo@yahoo.com">ozzzo@yahoo.com</a>>                wrote:<br clear="none">                ><br clear="none">                ><br clear="none">                > Following [1] I successfully set                "amqp_durable_queues = True" and restricted HA to the                appropriate queues, but I'm having trouble with some of                the other settings such as "expires" and "message-ttl".                Does anyone have an example of a working kolla config                that includes these changes?<br clear="none">                ><br clear="none">                > [1] <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit">https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit</a><br clear="none">                > On Monday, December 13, 2021, 07:51:32 AM EST,                Herve Beraud <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:hberaud@redhat.com" target="_blank" href="mailto:hberaud@redhat.com">hberaud@redhat.com</a>>                wrote:<br clear="none">                ><br clear="none">                ><br clear="none">                > So, your config snippet LGTM.<br clear="none">                ><br clear="none">                > Le ven. 10 déc. 2021 à 17:50, Albert Braden <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:ozzzo@yahoo.com" target="_blank" href="mailto:ozzzo@yahoo.com">ozzzo@yahoo.com</a>>                a écrit :<br clear="none">                ><br clear="none">                > Sorry, that was a transcription error. I thought                "True" and my fingers typed "False." The correct lines                are:<br clear="none">                ><br clear="none">                > [oslo_messaging_rabbit]<br clear="none">                > amqp_durable_queues = True<br clear="none">                ><br clear="none">                > On Friday, December 10, 2021, 02:55:55 AM EST,                Herve Beraud <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:hberaud@redhat.com" target="_blank" href="mailto:hberaud@redhat.com">hberaud@redhat.com</a>>                wrote:<br clear="none">                ><br clear="none">                ><br clear="none">                > If you plan to let `amqp_durable_queues = False`                (i.e if you plan to keep this config equal to false),                then you don't need to add these config lines as this is                already the default value [1].<br clear="none">                ><br clear="none">                > [1] <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messaging/_drivers/amqp.py#L34">https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messaging/_drivers/amqp.py#L34</a><br clear="none">                ><br clear="none">                > Le jeu. 9 déc. 2021 à 22:40, Albert Braden <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:ozzzo@yahoo.com" target="_blank" href="mailto:ozzzo@yahoo.com">ozzzo@yahoo.com</a>>                a écrit :<br clear="none">                ><br clear="none">                > Replying from my home email because I've been asked                to not email the list from my work email anymore, until                I get permission from upper management.<br clear="none">                ><br clear="none">                > I'm not sure I follow. I was planning to add 2                lines to etc/kolla/config/global.conf:<br clear="none">                ><br clear="none">                > [oslo_messaging_rabbit]<br clear="none">                > amqp_durable_queues = False<br clear="none">                ><br clear="none">                > Is that not sufficient? What is involved in                configuring dedicated control exchanges for each                service? What would that look like in the config?<br clear="none">                ><br clear="none">                ><br clear="none">                > From: Herve Beraud <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:hberaud@redhat.com" target="_blank" href="mailto:hberaud@redhat.com">hberaud@redhat.com</a>><br clear="none">                > Sent: Thursday, December 9, 2021 2:45 AM<br clear="none">                > To: Bogdan Dobrelya <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:bdobreli@redhat.com" target="_blank" href="mailto:bdobreli@redhat.com">bdobreli@redhat.com</a>><br clear="none">                > Cc: <a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:openstack-discuss@lists.openstack.org" target="_blank" href="mailto:openstack-discuss@lists.openstack.org">openstack-discuss@lists.openstack.org</a><br clear="none">                > Subject: [EXTERNAL] Re: [ops] [kolla] RabbitMQ High                Availability<br clear="none">                ><br clear="none">                ><br clear="none">                ><br clear="none">                > Caution: This email originated from outside the                organization. Do not click links or open attachments                unless you recognize the sender and know the content is                safe.<br clear="none">                ><br clear="none">                ><br clear="none">                ><br clear="none">                ><br clear="none">                ><br clear="none">                > Le mer. 8 déc. 2021 à 11:48, Bogdan Dobrelya <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:bdobreli@redhat.com" target="_blank" href="mailto:bdobreli@redhat.com">bdobreli@redhat.com</a>> a                écrit :<br clear="none">                ><br clear="none">                > Please see inline<br clear="none">                ><br clear="none">                > >> I read this with great interest because we                are seeing this issue. Questions:<br clear="none">                > >><br clear="none">                > >> 1. We are running kola-ansible Train, and                our RMQ version is 3.7.23. Should we be upgrading our                Train clusters to use 3.8.x?<br clear="none">                > >> 2. Document [2] recommends policy                '^(?!(amq\.)|(.*_fanout_)|(reply_)).*'. I don't see this                in our ansible playbooks, nor in any of the config files                in the RMQ container. What would this look like in                Ansible, and what should the resulting container config                look like?<br clear="none">                > >> 3. It appears that we are not setting                "amqp_durable_queues = True". What does this setting                look like in Ansible, and what file does it go into?<br clear="none">                > ><br clear="none">                > > Note that even having rabbit HA policies                adjusted like that and its HA<br clear="none">                > > replication factor [0] decreased (e.g. to a                2), there still might be<br clear="none">                > > high churn caused by a large enough number of                replicated durable RPC<br clear="none">                > > topic queues. And that might cripple the cloud                down with the incurred<br clear="none">                > > I/O overhead because a durable queue requires                all messages in it to be<br clear="none">                > > persisted to a disk (for all the messaging                cluster replicas) before they<br clear="none">                > > are ack'ed by the broker.<br clear="none">                > ><br clear="none">                > > Given that said, Oslo messaging would likely                require a more granular<br clear="none">                > > control for topic exchanges and the durable                queues flag - to tell it to<br clear="none">                > > declare as durable only the most critical                paths of a service. A single<br clear="none">                > > config setting and a single control exchange                per a service might be not<br clear="none">                > > enough.<br clear="none">                ><br clear="none">                > Also note that therefore, amqp_durable_queue=True                requires dedicated<br clear="none">                > control exchanges configured for each service.                Those that use<br clear="none">                > 'openstack' as a default cannot turn the feature                ON. Changing it to a<br clear="none">                > service specific might also cause upgrade impact,                as described in the<br clear="none">                > topic [3].<br clear="none">                ><br clear="none">                ><br clear="none">                ><br clear="none">                > The same is true for `amqp_auto_delete=True`. That                requires dedicated control exchanges else it won't work                if each service defines its own policy on a shared                control exchange (e.g `openstack`) and if policies                differ from each other.<br clear="none">                ><br clear="none">                ><br clear="none">                ><br clear="none">                > [3] <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://review.opendev.org/q/topic:scope-config-opts">https://review.opendev.org/q/topic:scope-config-opts</a><br clear="none">                ><br clear="none">                > ><br clear="none">                > > There are also race conditions with durable                queues enabled, like [1]. A<br clear="none">                > > solution could be where each service declare                its own dedicated control<br clear="none">                > > exchange with its own configuration.<br clear="none">                > ><br clear="none">                > > Finally, openstack components should add                perhaps a *.next CI job to test<br clear="none">                > > it with durable queues, like [2]<br clear="none">                > ><br clear="none">                > > [0] <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://www.rabbitmq.com/ha.html#replication-factor">https://www.rabbitmq.com/ha.html#replication-factor</a><br clear="none">                > ><br clear="none">                > > [1]<br clear="none">                > > <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://zuul.opendev.org/t/openstack/build/aa514dd788f34cc1be3800e6d7dba0e8/log/controller/logs/screen-n-cpu.txt">https://zuul.opendev.org/t/openstack/build/aa514dd788f34cc1be3800e6d7dba0e8/log/controller/logs/screen-n-cpu.txt</a><br clear="none">                > ><br clear="none">                > > [2] <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://review.opendev.org/c/openstack/nova/+/820523">https://review.opendev.org/c/openstack/nova/+/820523</a><br clear="none">                > ><br clear="none">                > >><br clear="none">                > >> Does anyone have a sample set of RMQ                config files that they can share?<br clear="none">                > >><br clear="none">                > >> It looks like my Outlook has ruined the                link; reposting:<br clear="none">                > >> [2] <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit">https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit</a><br clear="none">                > ><br clear="none">                > ><br clear="none">                > > --<br clear="none">                > > Best regards,<br clear="none">                > > Bogdan Dobrelya,<br clear="none">                > > Irc #bogdando<br clear="none">                ><br clear="none">                ><br clear="none">                > --<br clear="none">                > Best regards,<br clear="none">                > Bogdan Dobrelya,<br clear="none">                > Irc #bogdando<br clear="none">                ><br clear="none">                ><br clear="none">                ><br clear="none">                ><br clear="none">                > --<br clear="none">                ><br clear="none">                > Hervé Beraud<br clear="none">                ><br clear="none">                > Senior Software Engineer at Red Hat<br clear="none">                ><br clear="none">                > irc: hberaud<br clear="none">                ><br clear="none">                > <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://github.com/4383/">https://github.com/4383/</a><br clear="none">                ><br clear="none">                > <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://twitter.com/4383hberaud">https://twitter.com/4383hberaud</a><br clear="none">                ><br clear="none">                ><br clear="none">                ><br clear="none">                > --<br clear="none">                > Hervé Beraud<br clear="none">                > Senior Software Engineer at Red Hat<br clear="none">                > irc: hberaud<br clear="none">                > <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://github.com/4383/">https://github.com/4383/</a><br clear="none">                > <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://twitter.com/4383hberaud">https://twitter.com/4383hberaud</a><br clear="none">                ><br clear="none">                ><br clear="none">                ><br clear="none">                > --<br clear="none">                > Hervé Beraud<br clear="none">                > Senior Software Engineer at Red Hat<br clear="none">                > irc: hberaud<br clear="none">                > <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://github.com/4383/">https://github.com/4383/</a><br clear="none">                > <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://twitter.com/4383hberaud">https://twitter.com/4383hberaud</a><br clear="none">                ><br clear="none">              </div>            </div>          </div>        </div>      </div>      </div></div></div></div>            </div>                </div></div></div></div></div>            </div>                </div></div></div></div></div>            </div>                </div>