In an attempt to simplify I tried this: kolla-ansible stop --tags nova verified `docker ps | grep nova` returns nothing on all deply nodes ran `kolla-ansible genconfig --tags nova` verified it makes no changes ran `kolla-ansible reconfigure --tags rabbitmq` which only made a single trivial "change": TASK [rabbitmq : Get new RabbitMQ version] ************************* changed: [control-0] ran `kolla-ansible rabbitmq-reset-state` to flush all the queues, no errors reported verified `docker ps | grep nova` still returns nothing on all deply nodes manually started one `nova-conductor`: root@control-0:/# docker start nova_conductor nova_conductor Now right or wrong I would expect a single service to create the queue in whatever way it wants and be self consistent but I immediately get the `PRECONDITION_FAILED - invalid property 'non-durable' for queue 'conductor' in vhost '/'` error. Inside the nova-conductor container in nova.conf I see: [oslo_messaging_rabbit] use_queue_manager = true heartbeat_in_pthread = false rabbit_quorum_queue = true rabbit_stream_fanout = true rabbit_qos_prefetch_count = 1 rabbit_transient_quorum_queue = true I don't know what I should expect there but the "transient" seem suspect given the documentation about switching to durable queues ... Kind of grasping straws at this point, hopefully the error of my ways is clearer to others than it is to me. Thanks, -Jon On Mon, Oct 27, 2025 at 05:19:06PM -0400, Jonathan Proulx wrote: :On Mon, Oct 27, 2025 at 09:29:01AM +0100, Bartosz Bezak wrote: ::Hi Jonathan, :: ::Please double check if your rabbitmq settings are correct for epoxy release - https://docs.openstack.org/kolla-ansible/2025.1/reference/message-queues/rab... : :That is what I looked at first. : :$ grep om_ /etc/kolla/globals.yml : :om_enable_queue_manager: true :om_enable_rabbitmq_quorum_queues: true :om_enable_rabbitmq_transient_quorum_queue: true :om_enable_rabbitmq_stream_fanout: true : :This matches what is documented there I belive. : :On my first attempt I went through that update procedure. When it :failed I did clean OS reinstalls of the target systems. Those values :were set before the initial deploy on the clean systems. : :Just now I steped through again to verify. Unsurprisingly since the :values in global.yml dind't change when I ran 'kolla-ansible :genconfig' nothing changed either. : :kolla-ansible reconfigure --tags rabbitmq :also made no changes : :kolla-ansible rabbitmq-reset-state :obviously does reset state : :then when I re-deploy the stopped service I still see the exception, :for example in nova-conductor: : :2025-10-27 17:13:15.831 3313 ERROR oslo_service.backend.eventlet.service oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 128.30.3.147:5672 after inf tries: Queue.declare: (406) PRECONDITION_FAILED - invalid property 'non-durable' for queue 'conductor' in vhost '/' :2025-10-27 17:13:15.831 3313 ERROR oslo_service.backend.eventlet.service : :It certianly *feels* like that is the issue, but I can't seem to find :where it's going wrong. : :-Jon : :-- :Jonathan Proulx (he/him) :Sr. Technical Architect :The Infrastructure Group :MIT CSAIL : :: :: ::Best regards, ::Bartosz Bezak :: :: :: :: ::> On 24 Oct 2025, at 18:29, Jonathan Proulx <jon@csail.mit.edu> wrote: ::> ::> Hi All, ::> ::> I'm putting "fresh" in quotes becuase I expect the issue is some home ::> my deploy isn't as fresh as I belive it to be. ::> ::> I did have and older test environment which was serveral steps behind ::> (and not quite workign for other reasons). ::> ::> The presenting issues is this error (and other similar ones for other ::> services): ::> ::> 2025-10-24 11:44:07.840 1076 WARNING oslo.messaging._drivers.impl_rabbit [None req-4eff1f78-c16a-4bc9-b0da-9ce0a80b66ba - - - - - -] [83941886-4e9c-4d0c-b3b0-53c107f018e9] Queue conductor could not be declared probably because of conflicting configurations: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'true' but current is 'false': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'true' but current is 'false' ::> 2025-10-24 11:44:07.841 1076 INFO oslo.messaging._drivers.impl_rabbit [None req-4eff1f78-c16a-4bc9-b0da-9ce0a80b66ba - - - - - -] [83941886-4e9c-4d0c-b3b0-53c107f018e9] Retrying to declare the exchange (nova) as non durable ::> ::> I'm puzzled since the services are deploying to recently reinstalled ::> nodes (I'm sure that part is fresh) how the queues can be created with ::> flags the services don't expect. ::> ::> my shell history tells me I'm using these versions (in a venev): ::> ::> pip3 install --upgrade git+https://opendev.org/openstack/kolla-ansible@stable/2025.1 ::> pip3 install --upgrade git+https://opendev.org/openstack/kolla@stable/2025.1 ::> pip3 install --upgrade 'ansible-core>=2.17,<2.18.99' ::> ::> The only bits I know are left over and "upgraded" from previous attempts are my configs in /etc/kolla. ::> ::> My deploy host wasn't reinstalled but I did delete and recreate the venv. Using Ubuntu 24.04 as the base OS ::> ::> Where do I go from here to find my wrong turn? ::> ::> Thanks, ::> -Jon ::> ::> -- ::> Jonathan "Some Day I'll Upgrade From Mitaka" Proulx (he/him) ::> Sr. Technical Architect ::> The Infrastructure Group ::> MIT CSAIL :: -- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL