[kolla-ansible] Rabbitmq queue creation issues in "fresh" 2025.1 deploy
Hi All, I'm putting "fresh" in quotes becuase I expect the issue is some home my deploy isn't as fresh as I belive it to be. I did have and older test environment which was serveral steps behind (and not quite workign for other reasons). The presenting issues is this error (and other similar ones for other services): 2025-10-24 11:44:07.840 1076 WARNING oslo.messaging._drivers.impl_rabbit [None req-4eff1f78-c16a-4bc9-b0da-9ce0a80b66ba - - - - - -] [83941886-4e9c-4d0c-b3b0-53c107f018e9] Queue conductor could not be declared probably because of conflicting configurations: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'true' but current is 'false': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'true' but current is 'false' 2025-10-24 11:44:07.841 1076 INFO oslo.messaging._drivers.impl_rabbit [None req-4eff1f78-c16a-4bc9-b0da-9ce0a80b66ba - - - - - -] [83941886-4e9c-4d0c-b3b0-53c107f018e9] Retrying to declare the exchange (nova) as non durable I'm puzzled since the services are deploying to recently reinstalled nodes (I'm sure that part is fresh) how the queues can be created with flags the services don't expect. my shell history tells me I'm using these versions (in a venev): pip3 install --upgrade git+https://opendev.org/openstack/kolla-ansible@stable/2025.1 pip3 install --upgrade git+https://opendev.org/openstack/kolla@stable/2025.1 pip3 install --upgrade 'ansible-core>=2.17,<2.18.99' The only bits I know are left over and "upgraded" from previous attempts are my configs in /etc/kolla. My deploy host wasn't reinstalled but I did delete and recreate the venv. Using Ubuntu 24.04 as the base OS Where do I go from here to find my wrong turn? Thanks, -Jon -- Jonathan "Some Day I'll Upgrade From Mitaka" Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL
Hi Jonathan, Please double check if your rabbitmq settings are correct for epoxy release - https://docs.openstack.org/kolla-ansible/2025.1/reference/message-queues/rab... Best regards, Bartosz Bezak
On 24 Oct 2025, at 18:29, Jonathan Proulx <jon@csail.mit.edu> wrote:
Hi All,
I'm putting "fresh" in quotes becuase I expect the issue is some home my deploy isn't as fresh as I belive it to be.
I did have and older test environment which was serveral steps behind (and not quite workign for other reasons).
The presenting issues is this error (and other similar ones for other services):
2025-10-24 11:44:07.840 1076 WARNING oslo.messaging._drivers.impl_rabbit [None req-4eff1f78-c16a-4bc9-b0da-9ce0a80b66ba - - - - - -] [83941886-4e9c-4d0c-b3b0-53c107f018e9] Queue conductor could not be declared probably because of conflicting configurations: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'true' but current is 'false': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'true' but current is 'false' 2025-10-24 11:44:07.841 1076 INFO oslo.messaging._drivers.impl_rabbit [None req-4eff1f78-c16a-4bc9-b0da-9ce0a80b66ba - - - - - -] [83941886-4e9c-4d0c-b3b0-53c107f018e9] Retrying to declare the exchange (nova) as non durable
I'm puzzled since the services are deploying to recently reinstalled nodes (I'm sure that part is fresh) how the queues can be created with flags the services don't expect.
my shell history tells me I'm using these versions (in a venev):
pip3 install --upgrade git+https://opendev.org/openstack/kolla-ansible@stable/2025.1 pip3 install --upgrade git+https://opendev.org/openstack/kolla@stable/2025.1 pip3 install --upgrade 'ansible-core>=2.17,<2.18.99'
The only bits I know are left over and "upgraded" from previous attempts are my configs in /etc/kolla.
My deploy host wasn't reinstalled but I did delete and recreate the venv. Using Ubuntu 24.04 as the base OS
Where do I go from here to find my wrong turn?
Thanks, -Jon
-- Jonathan "Some Day I'll Upgrade From Mitaka" Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL
On Mon, Oct 27, 2025 at 09:29:01AM +0100, Bartosz Bezak wrote: :Hi Jonathan, : :Please double check if your rabbitmq settings are correct for epoxy release - https://docs.openstack.org/kolla-ansible/2025.1/reference/message-queues/rab... That is what I looked at first. $ grep om_ /etc/kolla/globals.yml om_enable_queue_manager: true om_enable_rabbitmq_quorum_queues: true om_enable_rabbitmq_transient_quorum_queue: true om_enable_rabbitmq_stream_fanout: true This matches what is documented there I belive. On my first attempt I went through that update procedure. When it failed I did clean OS reinstalls of the target systems. Those values were set before the initial deploy on the clean systems. Just now I steped through again to verify. Unsurprisingly since the values in global.yml dind't change when I ran 'kolla-ansible genconfig' nothing changed either. kolla-ansible reconfigure --tags rabbitmq also made no changes kolla-ansible rabbitmq-reset-state obviously does reset state then when I re-deploy the stopped service I still see the exception, for example in nova-conductor: 2025-10-27 17:13:15.831 3313 ERROR oslo_service.backend.eventlet.service oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 128.30.3.147:5672 after inf tries: Queue.declare: (406) PRECONDITION_FAILED - invalid property 'non-durable' for queue 'conductor' in vhost '/' 2025-10-27 17:13:15.831 3313 ERROR oslo_service.backend.eventlet.service It certianly *feels* like that is the issue, but I can't seem to find where it's going wrong. -Jon -- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL : : :Best regards, :Bartosz Bezak : : : : :> On 24 Oct 2025, at 18:29, Jonathan Proulx <jon@csail.mit.edu> wrote: :> :> Hi All, :> :> I'm putting "fresh" in quotes becuase I expect the issue is some home :> my deploy isn't as fresh as I belive it to be. :> :> I did have and older test environment which was serveral steps behind :> (and not quite workign for other reasons). :> :> The presenting issues is this error (and other similar ones for other :> services): :> :> 2025-10-24 11:44:07.840 1076 WARNING oslo.messaging._drivers.impl_rabbit [None req-4eff1f78-c16a-4bc9-b0da-9ce0a80b66ba - - - - - -] [83941886-4e9c-4d0c-b3b0-53c107f018e9] Queue conductor could not be declared probably because of conflicting configurations: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'true' but current is 'false': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'true' but current is 'false' :> 2025-10-24 11:44:07.841 1076 INFO oslo.messaging._drivers.impl_rabbit [None req-4eff1f78-c16a-4bc9-b0da-9ce0a80b66ba - - - - - -] [83941886-4e9c-4d0c-b3b0-53c107f018e9] Retrying to declare the exchange (nova) as non durable :> :> I'm puzzled since the services are deploying to recently reinstalled :> nodes (I'm sure that part is fresh) how the queues can be created with :> flags the services don't expect. :> :> my shell history tells me I'm using these versions (in a venev): :> :> pip3 install --upgrade git+https://opendev.org/openstack/kolla-ansible@stable/2025.1 :> pip3 install --upgrade git+https://opendev.org/openstack/kolla@stable/2025.1 :> pip3 install --upgrade 'ansible-core>=2.17,<2.18.99' :> :> The only bits I know are left over and "upgraded" from previous attempts are my configs in /etc/kolla. :> :> My deploy host wasn't reinstalled but I did delete and recreate the venv. Using Ubuntu 24.04 as the base OS :> :> Where do I go from here to find my wrong turn? :> :> Thanks, :> -Jon :> :> -- :> Jonathan "Some Day I'll Upgrade From Mitaka" Proulx (he/him) :> Sr. Technical Architect :> The Infrastructure Group :> MIT CSAIL :
In an attempt to simplify I tried this: kolla-ansible stop --tags nova verified `docker ps | grep nova` returns nothing on all deply nodes ran `kolla-ansible genconfig --tags nova` verified it makes no changes ran `kolla-ansible reconfigure --tags rabbitmq` which only made a single trivial "change": TASK [rabbitmq : Get new RabbitMQ version] ************************* changed: [control-0] ran `kolla-ansible rabbitmq-reset-state` to flush all the queues, no errors reported verified `docker ps | grep nova` still returns nothing on all deply nodes manually started one `nova-conductor`: root@control-0:/# docker start nova_conductor nova_conductor Now right or wrong I would expect a single service to create the queue in whatever way it wants and be self consistent but I immediately get the `PRECONDITION_FAILED - invalid property 'non-durable' for queue 'conductor' in vhost '/'` error. Inside the nova-conductor container in nova.conf I see: [oslo_messaging_rabbit] use_queue_manager = true heartbeat_in_pthread = false rabbit_quorum_queue = true rabbit_stream_fanout = true rabbit_qos_prefetch_count = 1 rabbit_transient_quorum_queue = true I don't know what I should expect there but the "transient" seem suspect given the documentation about switching to durable queues ... Kind of grasping straws at this point, hopefully the error of my ways is clearer to others than it is to me. Thanks, -Jon On Mon, Oct 27, 2025 at 05:19:06PM -0400, Jonathan Proulx wrote: :On Mon, Oct 27, 2025 at 09:29:01AM +0100, Bartosz Bezak wrote: ::Hi Jonathan, :: ::Please double check if your rabbitmq settings are correct for epoxy release - https://docs.openstack.org/kolla-ansible/2025.1/reference/message-queues/rab... : :That is what I looked at first. : :$ grep om_ /etc/kolla/globals.yml : :om_enable_queue_manager: true :om_enable_rabbitmq_quorum_queues: true :om_enable_rabbitmq_transient_quorum_queue: true :om_enable_rabbitmq_stream_fanout: true : :This matches what is documented there I belive. : :On my first attempt I went through that update procedure. When it :failed I did clean OS reinstalls of the target systems. Those values :were set before the initial deploy on the clean systems. : :Just now I steped through again to verify. Unsurprisingly since the :values in global.yml dind't change when I ran 'kolla-ansible :genconfig' nothing changed either. : :kolla-ansible reconfigure --tags rabbitmq :also made no changes : :kolla-ansible rabbitmq-reset-state :obviously does reset state : :then when I re-deploy the stopped service I still see the exception, :for example in nova-conductor: : :2025-10-27 17:13:15.831 3313 ERROR oslo_service.backend.eventlet.service oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 128.30.3.147:5672 after inf tries: Queue.declare: (406) PRECONDITION_FAILED - invalid property 'non-durable' for queue 'conductor' in vhost '/' :2025-10-27 17:13:15.831 3313 ERROR oslo_service.backend.eventlet.service : :It certianly *feels* like that is the issue, but I can't seem to find :where it's going wrong. : :-Jon : :-- :Jonathan Proulx (he/him) :Sr. Technical Architect :The Infrastructure Group :MIT CSAIL : :: :: ::Best regards, ::Bartosz Bezak :: :: :: :: ::> On 24 Oct 2025, at 18:29, Jonathan Proulx <jon@csail.mit.edu> wrote: ::> ::> Hi All, ::> ::> I'm putting "fresh" in quotes becuase I expect the issue is some home ::> my deploy isn't as fresh as I belive it to be. ::> ::> I did have and older test environment which was serveral steps behind ::> (and not quite workign for other reasons). ::> ::> The presenting issues is this error (and other similar ones for other ::> services): ::> ::> 2025-10-24 11:44:07.840 1076 WARNING oslo.messaging._drivers.impl_rabbit [None req-4eff1f78-c16a-4bc9-b0da-9ce0a80b66ba - - - - - -] [83941886-4e9c-4d0c-b3b0-53c107f018e9] Queue conductor could not be declared probably because of conflicting configurations: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'true' but current is 'false': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'true' but current is 'false' ::> 2025-10-24 11:44:07.841 1076 INFO oslo.messaging._drivers.impl_rabbit [None req-4eff1f78-c16a-4bc9-b0da-9ce0a80b66ba - - - - - -] [83941886-4e9c-4d0c-b3b0-53c107f018e9] Retrying to declare the exchange (nova) as non durable ::> ::> I'm puzzled since the services are deploying to recently reinstalled ::> nodes (I'm sure that part is fresh) how the queues can be created with ::> flags the services don't expect. ::> ::> my shell history tells me I'm using these versions (in a venev): ::> ::> pip3 install --upgrade git+https://opendev.org/openstack/kolla-ansible@stable/2025.1 ::> pip3 install --upgrade git+https://opendev.org/openstack/kolla@stable/2025.1 ::> pip3 install --upgrade 'ansible-core>=2.17,<2.18.99' ::> ::> The only bits I know are left over and "upgraded" from previous attempts are my configs in /etc/kolla. ::> ::> My deploy host wasn't reinstalled but I did delete and recreate the venv. Using Ubuntu 24.04 as the base OS ::> ::> Where do I go from here to find my wrong turn? ::> ::> Thanks, ::> -Jon ::> ::> -- ::> Jonathan "Some Day I'll Upgrade From Mitaka" Proulx (he/him) ::> Sr. Technical Architect ::> The Infrastructure Group ::> MIT CSAIL :: -- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL
Thank you all for playing along... I haven't proved this is the issue but clearly this is the issue: I did cleanly reinstall the set of systems I meant to be testing on, but apparently I had *previously* been testing on a larger subset so there are old services which are no longer under active kolla-ansible management throwing bad bits at my rabbitmq Oops :) -Jon -- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL
participants (2)
-
Bartosz Bezak
-
Jonathan Proulx