Hi, To tell the truth I never seen such issue, but one hint what to check. The amqp_durable_queues setting is coming from oslo.messaging (see [1]) by default it is false, so one thing should be to check if all your services has this default, or accidentally one of them set it to true. You can see in the logs these values when the services are started, if they are not directly set in your Neutron config files. [1]: https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messagin... Best wishes Lajos Katona Manohar Kundapura (makundap) <makundap@cisco.com> ezt írta (időpont: 2025. márc. 6., Cs, 17:30):
Hi Everyone,
I have the 2024.1 Caracal multi node setup working. Now I am trying to enable SRIOV on this setup by following the below link,
https://docs.openstack.org/neutron/2024.1/admin/config-sriov.html
Observing the below issue while running the “neutron-sriov-agent” service on compute node and it is restarting on every 2 seconds.
Kindly let me know if I need to enable any configs specific to “neutron-sriov-agent” service.
*# tail -f /var/log/neutron/neutron-sriov-agent.log*
*2025-03-06 15:02:42.968 9311 INFO neutron.common.config [-] Logging enabled!*
*2025-03-06 15:02:42.969 9311 INFO neutron.common.config [-] /usr/bin/neutron-sriov-nic-agent version 24.0.0*
*2025-03-06 15:02:42.969 9311 DEBUG neutron.common.config [-] command line: /usr/bin/neutron-sriov-nic-agent --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/sriov_agent.ini --log-file=/var/log/neutron/neutron-sriov-agent.log setup_logging /usr/lib/python3/dist-packages/neutron/common/config.py:123*
*2025-03-06 15:02:42.970 9311 DEBUG neutron.agent.common.utils [-] Std library socket module is patched by eventlet. Requesting std library socket module from eventlet. __init__ /usr/lib/python3/dist-packages/neutron/agent/common/utils.py:91*
*2025-03-06 15:02:42.971 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Physical Devices mappings: {'vf_inside': ['enp216s0f0'], 'vf_outside': ['enp216s0f1']}*
*2025-03-06 15:02:42.971 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Exclude Devices: {}*
*2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider bandwidths: {}*
*2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider inventory defaults: {'allocation_ratio': 1.0, 'min_unit': 1, 'step_size': 1, 'reserved': 0}*
*2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider hypervisors: {'enp216s0f1': 'u36c01p06-ostest-cmp2.openstack.local', 'enp216s0f0': 'u36c01p06-ostest-cmp2.openstack.local'}*
*2025-03-06 15:02:42.973 9311 DEBUG neutron.plugins.ml2.drivers.mech_sriov.agent.eswitch_manager [-] Number of VFs configured on device enp216s0f0: 8 get_numvfs /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py:134*
*2025-03-06 15:02:42.973 9311 DEBUG neutron.plugins.ml2.drivers.mech_sriov.agent.eswitch_manager [-] Number of VFs configured on device enp216s0f1: 8 get_numvfs /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py:134*
*2025-03-06 15:02:42.976 9311 DEBUG neutron.agent.securitygroups_rpc [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Init firewall settings (driver=noop) init_firewall /usr/lib/python3/dist-packages/neutron/agent/securitygroups_rpc.py:124*
*2025-03-06 15:02:42.976 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] RPC agent_id: nic-switch-agent.u36c01p06-ostest-cmp2*
*2025-03-06 15:02:42.981 9311 INFO neutron.agent.agent_extensions_manager [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Loaded agent extensions: []*
*2025-03-06 15:02:43.037 9311 WARNING oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'*
*2025-03-06 15:02:43.038 9311 INFO oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] [7d2d8221-cd72-4203-b717-46c9d4fac7a9] Retrying to declare the exchange (q-agent-notifier-port-update_fanout) as non durable*
*2025-03-06 15:02:43.041 9311 ERROR oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Failed to declare consumer for topic 'q-agent-notifier-port-update': Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'*
*2025-03-06 15:02:43.046 9311 ERROR oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Unable to connect to AMQP server on 10.10.4.19:5671 <http://10.10.4.19:5671> after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'*
*2025-03-06 15:02:43.047 9311 CRITICAL neutron [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Unhandled error: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.10.4.19:5671 <http://10.10.4.19:5671> after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'*
*2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last):*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 421, in declare*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.queue.declare()*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 605, in declare*
*2025-03-06 15:02:43.047 9311 ERROR neutron self._create_exchange(nowait=nowait, channel=channel)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 612, in _create_exchange*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.exchange.declare(nowait=nowait, channel=channel)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 180, in declare*
*2025-03-06 15:02:43.047 9311 ERROR neutron return (channel or self.channel).exchange_declare(*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 612, in exchange_declare*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.send_method(*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 66, in send_method*
*2025-03-06 15:02:43.047 9311 ERROR neutron return self.wait(wait, returns_tuple=returns_tuple)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 86, in wait*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.drain_events(timeout=timeout)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 522, in drain_events*
*2025-03-06 15:02:43.047 9311 ERROR neutron while not self.blocking_read(timeout):*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 528, in blocking_read*
*2025-03-06 15:02:43.047 9311 ERROR neutron return self.on_inbound_frame(frame)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/method_framing.py", line 53, in on_frame*
*2025-03-06 15:02:43.047 9311 ERROR neutron callback(channel, method_sig, buf, None)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 534, in on_inbound_method*
*2025-03-06 15:02:43.047 9311 ERROR neutron return self.channels[channel_id].dispatch_method(*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 143, in dispatch_method*
*2025-03-06 15:02:43.047 9311 ERROR neutron listener(*args)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 281, in _on_close*
*2025-03-06 15:02:43.047 9311 ERROR neutron raise error_for_code(*
*2025-03-06 15:02:43.047 9311 ERROR neutron amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'*
*2025-03-06 15:02:43.047 9311 ERROR neutron *
*2025-03-06 15:02:43.047 9311 ERROR neutron During handling of the above exception, another exception occurred:*
*2025-03-06 15:02:43.047 9311 ERROR neutron *
*2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last):*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1036, in ensure*
*2025-03-06 15:02:43.047 9311 ERROR neutron ret, channel = autoretry_method()*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/connection.py", line 524, in _ensured*
*2025-03-06 15:02:43.047 9311 ERROR neutron return fun(*args, **kwargs)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/connection.py", line 600, in __call__*
*2025-03-06 15:02:43.047 9311 ERROR neutron return fun(*args, channel=channels[0], **kwargs), channels[0]*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1025, in execute_method*
*2025-03-06 15:02:43.047 9311 ERROR neutron method()*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1262, in _declare_consumer*
*2025-03-06 15:02:43.047 9311 ERROR neutron consumer.declare(self)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 431, in declare*
*2025-03-06 15:02:43.047 9311 ERROR neutron self._declare_fallback(err, conn, consumer_arguments)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 396, in _declare_fallback*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.queue.declare()*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 605, in declare*
*2025-03-06 15:02:43.047 9311 ERROR neutron self._create_exchange(nowait=nowait, channel=channel)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 612, in _create_exchange*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.exchange.declare(nowait=nowait, channel=channel)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 180, in declare*
*2025-03-06 15:02:43.047 9311 ERROR neutron return (channel or self.channel).exchange_declare(*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 612, in exchange_declare*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.send_method(*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 66, in send_method*
*2025-03-06 15:02:43.047 9311 ERROR neutron return self.wait(wait, returns_tuple=returns_tuple)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 86, in wait*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.drain_events(timeout=timeout)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 522, in drain_events*
*2025-03-06 15:02:43.047 9311 ERROR neutron while not self.blocking_read(timeout):*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 528, in blocking_read*
*2025-03-06 15:02:43.047 9311 ERROR neutron return self.on_inbound_frame(frame)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/method_framing.py", line 53, in on_frame*
*2025-03-06 15:02:43.047 9311 ERROR neutron callback(channel, method_sig, buf, None)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 534, in on_inbound_method*
*2025-03-06 15:02:43.047 9311 ERROR neutron return self.channels[channel_id].dispatch_method(*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 143, in dispatch_method*
*2025-03-06 15:02:43.047 9311 ERROR neutron listener(*args)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 281, in _on_close*
*2025-03-06 15:02:43.047 9311 ERROR neutron raise error_for_code(*
*2025-03-06 15:02:43.047 9311 ERROR neutron amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'*
*2025-03-06 15:02:43.047 9311 ERROR neutron *
*2025-03-06 15:02:43.047 9311 ERROR neutron During handling of the above exception, another exception occurred:*
*2025-03-06 15:02:43.047 9311 ERROR neutron *
*2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last):*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/bin/neutron-sriov-nic-agent", line 10, in <module>*
*2025-03-06 15:02:43.047 9311 ERROR neutron sys.exit(main())*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/cmd/eventlet/plugins/sriov_nic_neutron_agent.py", line 27, in main*
*2025-03-06 15:02:43.047 9311 ERROR neutron agent_main.main()*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 568, in main*
*2025-03-06 15:02:43.047 9311 ERROR neutron agent = SriovNicSwitchAgent(device_mappings,*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 181, in __init__*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.consume_in_threads()*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/rpc.py", line 345, in consume_in_threads*
*2025-03-06 15:02:43.047 9311 ERROR neutron server.start()*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 267, in wrapper*
*2025-03-06 15:02:43.047 9311 ERROR neutron states[state].run_once(lambda: fn(self, *args, **kwargs),*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 188, in run_once*
*2025-03-06 15:02:43.047 9311 ERROR neutron post_fn = fn()*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 267, in <lambda>*
*2025-03-06 15:02:43.047 9311 ERROR neutron states[state].run_once(lambda: fn(self, *args, **kwargs),*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 413, in start*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.listener = self._create_listener()*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 150, in _create_listener*
*2025-03-06 15:02:43.047 9311 ERROR neutron return self.transport._listen(self._target, 1, None)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 142, in _listen*
*2025-03-06 15:02:43.047 9311 ERROR neutron return self._driver.listen(target, batch_size,*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 709, in listen*
*2025-03-06 15:02:43.047 9311 ERROR neutron conn.declare_fanout_consumer(target.topic, listener)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1398, in declare_fanout_consumer*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.declare_consumer(consumer)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1273, in declare_consumer*
*2025-03-06 15:02:43.047 9311 ERROR neutron return self.ensure(_declare_consumer,*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1058, in ensure*
*2025-03-06 15:02:43.047 9311 ERROR neutron raise exceptions.MessageDeliveryFailure(msg)*
*2025-03-06 15:02:43.047 9311 ERROR neutron oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.10.4.19:5671 <http://10.10.4.19:5671> after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'*
*2025-03-06 15:02:43.047 9311 ERROR neutron*
Thanks,
Manohar