Hi,
To tell the truth I never seen such issue, but one hint what to check.
The amqp_durable_queues setting is coming from oslo.messaging (see [1]) by default it is false, so one thing should be to check if all your services has this default, or accidentally one of them set it to true.
You can see in the logs these values when the services are started, if they are not directly set in your Neutron config files.

[1]: https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messaging/_drivers/amqp.py#L33

Best wishes
Lajos Katona

Manohar Kundapura (makundap) <makundap@cisco.com> ezt írta (időpont: 2025. márc. 6., Cs, 17:30):

Hi Everyone,

 

I have the 2024.1 Caracal multi node setup working. Now I am trying to enable SRIOV on this setup by following the below link,

https://docs.openstack.org/neutron/2024.1/admin/config-sriov.html

 

Observing the below issue while running the “neutron-sriov-agent” service on compute node and it is restarting on every 2 seconds.

Kindly let me know if I need to enable any configs specific to “neutron-sriov-agent” service.

 

# tail -f /var/log/neutron/neutron-sriov-agent.log

2025-03-06 15:02:42.968 9311 INFO neutron.common.config [-] Logging enabled!

2025-03-06 15:02:42.969 9311 INFO neutron.common.config [-] /usr/bin/neutron-sriov-nic-agent version 24.0.0

2025-03-06 15:02:42.969 9311 DEBUG neutron.common.config [-] command line: /usr/bin/neutron-sriov-nic-agent --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/sriov_agent.ini --log-file=/var/log/neutron/neutron-sriov-agent.log setup_logging /usr/lib/python3/dist-packages/neutron/common/config.py:123

2025-03-06 15:02:42.970 9311 DEBUG neutron.agent.common.utils [-] Std library socket module is patched by eventlet. Requesting std library socket module from eventlet. __init__ /usr/lib/python3/dist-packages/neutron/agent/common/utils.py:91

2025-03-06 15:02:42.971 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Physical Devices mappings: {'vf_inside': ['enp216s0f0'], 'vf_outside': ['enp216s0f1']}

2025-03-06 15:02:42.971 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Exclude Devices: {}

2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider bandwidths: {}

2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider inventory defaults: {'allocation_ratio': 1.0, 'min_unit': 1, 'step_size': 1, 'reserved': 0}

2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider hypervisors: {'enp216s0f1': 'u36c01p06-ostest-cmp2.openstack.local', 'enp216s0f0': 'u36c01p06-ostest-cmp2.openstack.local'}

2025-03-06 15:02:42.973 9311 DEBUG neutron.plugins.ml2.drivers.mech_sriov.agent.eswitch_manager [-] Number of VFs configured on device enp216s0f0: 8 get_numvfs /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py:134

2025-03-06 15:02:42.973 9311 DEBUG neutron.plugins.ml2.drivers.mech_sriov.agent.eswitch_manager [-] Number of VFs configured on device enp216s0f1: 8 get_numvfs /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py:134

2025-03-06 15:02:42.976 9311 DEBUG neutron.agent.securitygroups_rpc [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Init firewall settings (driver=noop) init_firewall /usr/lib/python3/dist-packages/neutron/agent/securitygroups_rpc.py:124

2025-03-06 15:02:42.976 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] RPC agent_id: nic-switch-agent.u36c01p06-ostest-cmp2

2025-03-06 15:02:42.981 9311 INFO neutron.agent.agent_extensions_manager [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Loaded agent extensions: []

2025-03-06 15:02:43.037 9311 WARNING oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'

2025-03-06 15:02:43.038 9311 INFO oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] [7d2d8221-cd72-4203-b717-46c9d4fac7a9] Retrying to declare the exchange (q-agent-notifier-port-update_fanout) as non durable

2025-03-06 15:02:43.041 9311 ERROR oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Failed to declare consumer for topic 'q-agent-notifier-port-update': Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'

2025-03-06 15:02:43.046 9311 ERROR oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Unable to connect to AMQP server on 10.10.4.19:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'

2025-03-06 15:02:43.047 9311 CRITICAL neutron [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Unhandled error: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.10.4.19:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'

2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last):

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 421, in declare

2025-03-06 15:02:43.047 9311 ERROR neutron     self.queue.declare()

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/kombu/entity.py", line 605, in declare

2025-03-06 15:02:43.047 9311 ERROR neutron     self._create_exchange(nowait=nowait, channel=channel)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/kombu/entity.py", line 612, in _create_exchange

2025-03-06 15:02:43.047 9311 ERROR neutron     self.exchange.declare(nowait=nowait, channel=channel)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/kombu/entity.py", line 180, in declare

2025-03-06 15:02:43.047 9311 ERROR neutron     return (channel or self.channel).exchange_declare(

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/amqp/channel.py", line 612, in exchange_declare

2025-03-06 15:02:43.047 9311 ERROR neutron     self.send_method(

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 66, in send_method

2025-03-06 15:02:43.047 9311 ERROR neutron     return self.wait(wait, returns_tuple=returns_tuple)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 86, in wait

2025-03-06 15:02:43.047 9311 ERROR neutron     self.connection.drain_events(timeout=timeout)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/amqp/connection.py", line 522, in drain_events

2025-03-06 15:02:43.047 9311 ERROR neutron     while not self.blocking_read(timeout):

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/amqp/connection.py", line 528, in blocking_read

2025-03-06 15:02:43.047 9311 ERROR neutron     return self.on_inbound_frame(frame)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/amqp/method_framing.py", line 53, in on_frame

2025-03-06 15:02:43.047 9311 ERROR neutron     callback(channel, method_sig, buf, None)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/amqp/connection.py", line 534, in on_inbound_method

2025-03-06 15:02:43.047 9311 ERROR neutron     return self.channels[channel_id].dispatch_method(

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 143, in dispatch_method

2025-03-06 15:02:43.047 9311 ERROR neutron     listener(*args)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/amqp/channel.py", line 281, in _on_close

2025-03-06 15:02:43.047 9311 ERROR neutron     raise error_for_code(

2025-03-06 15:02:43.047 9311 ERROR neutron amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'

2025-03-06 15:02:43.047 9311 ERROR neutron

2025-03-06 15:02:43.047 9311 ERROR neutron During handling of the above exception, another exception occurred:

2025-03-06 15:02:43.047 9311 ERROR neutron

2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last):

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1036, in ensure

2025-03-06 15:02:43.047 9311 ERROR neutron     ret, channel = autoretry_method()

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/kombu/connection.py", line 524, in _ensured

2025-03-06 15:02:43.047 9311 ERROR neutron     return fun(*args, **kwargs)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/kombu/connection.py", line 600, in __call__

2025-03-06 15:02:43.047 9311 ERROR neutron     return fun(*args, channel=channels[0], **kwargs), channels[0]

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1025, in execute_method

2025-03-06 15:02:43.047 9311 ERROR neutron     method()

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1262, in _declare_consumer

2025-03-06 15:02:43.047 9311 ERROR neutron     consumer.declare(self)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 431, in declare

2025-03-06 15:02:43.047 9311 ERROR neutron     self._declare_fallback(err, conn, consumer_arguments)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 396, in _declare_fallback

2025-03-06 15:02:43.047 9311 ERROR neutron     self.queue.declare()

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/kombu/entity.py", line 605, in declare

2025-03-06 15:02:43.047 9311 ERROR neutron     self._create_exchange(nowait=nowait, channel=channel)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/kombu/entity.py", line 612, in _create_exchange

2025-03-06 15:02:43.047 9311 ERROR neutron     self.exchange.declare(nowait=nowait, channel=channel)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/kombu/entity.py", line 180, in declare

2025-03-06 15:02:43.047 9311 ERROR neutron     return (channel or self.channel).exchange_declare(

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/amqp/channel.py", line 612, in exchange_declare

2025-03-06 15:02:43.047 9311 ERROR neutron     self.send_method(

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 66, in send_method

2025-03-06 15:02:43.047 9311 ERROR neutron     return self.wait(wait, returns_tuple=returns_tuple)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 86, in wait

2025-03-06 15:02:43.047 9311 ERROR neutron     self.connection.drain_events(timeout=timeout)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/amqp/connection.py", line 522, in drain_events

2025-03-06 15:02:43.047 9311 ERROR neutron     while not self.blocking_read(timeout):

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/amqp/connection.py", line 528, in blocking_read

2025-03-06 15:02:43.047 9311 ERROR neutron     return self.on_inbound_frame(frame)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/amqp/method_framing.py", line 53, in on_frame

2025-03-06 15:02:43.047 9311 ERROR neutron     callback(channel, method_sig, buf, None)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/amqp/connection.py", line 534, in on_inbound_method

2025-03-06 15:02:43.047 9311 ERROR neutron     return self.channels[channel_id].dispatch_method(

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 143, in dispatch_method

2025-03-06 15:02:43.047 9311 ERROR neutron     listener(*args)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/amqp/channel.py", line 281, in _on_close

2025-03-06 15:02:43.047 9311 ERROR neutron     raise error_for_code(

2025-03-06 15:02:43.047 9311 ERROR neutron amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'

2025-03-06 15:02:43.047 9311 ERROR neutron

2025-03-06 15:02:43.047 9311 ERROR neutron During handling of the above exception, another exception occurred:

2025-03-06 15:02:43.047 9311 ERROR neutron

2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last):

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/bin/neutron-sriov-nic-agent", line 10, in <module>

2025-03-06 15:02:43.047 9311 ERROR neutron     sys.exit(main())

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/neutron/cmd/eventlet/plugins/sriov_nic_neutron_agent.py", line 27, in main

2025-03-06 15:02:43.047 9311 ERROR neutron     agent_main.main()

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 568, in main

2025-03-06 15:02:43.047 9311 ERROR neutron     agent = SriovNicSwitchAgent(device_mappings,

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 181, in __init__

2025-03-06 15:02:43.047 9311 ERROR neutron     self.connection.consume_in_threads()

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/neutron_lib/rpc.py", line 345, in consume_in_threads

2025-03-06 15:02:43.047 9311 ERROR neutron     server.start()

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 267, in wrapper

2025-03-06 15:02:43.047 9311 ERROR neutron     states[state].run_once(lambda: fn(self, *args, **kwargs),

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 188, in run_once

2025-03-06 15:02:43.047 9311 ERROR neutron     post_fn = fn()

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 267, in <lambda>

2025-03-06 15:02:43.047 9311 ERROR neutron     states[state].run_once(lambda: fn(self, *args, **kwargs),

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 413, in start

2025-03-06 15:02:43.047 9311 ERROR neutron     self.listener = self._create_listener()

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 150, in _create_listener

2025-03-06 15:02:43.047 9311 ERROR neutron     return self.transport._listen(self._target, 1, None)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 142, in _listen

2025-03-06 15:02:43.047 9311 ERROR neutron     return self._driver.listen(target, batch_size,

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 709, in listen

2025-03-06 15:02:43.047 9311 ERROR neutron     conn.declare_fanout_consumer(target.topic, listener)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1398, in declare_fanout_consumer

2025-03-06 15:02:43.047 9311 ERROR neutron     self.declare_consumer(consumer)

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1273, in declare_consumer

2025-03-06 15:02:43.047 9311 ERROR neutron     return self.ensure(_declare_consumer,

2025-03-06 15:02:43.047 9311 ERROR neutron   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1058, in ensure

2025-03-06 15:02:43.047 9311 ERROR neutron     raise exceptions.MessageDeliveryFailure(msg)

2025-03-06 15:02:43.047 9311 ERROR neutron oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.10.4.19:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'

2025-03-06 15:02:43.047 9311 ERROR neutron

 

 

Thanks,

Manohar