Hi Everyone, I have the 2024.1 Caracal multi node setup working. Now I am trying to enable SRIOV on this setup by following the below link, https://docs.openstack.org/neutron/2024.1/admin/config-sriov.html Observing the below issue while running the “neutron-sriov-agent” service on compute node and it is restarting on every 2 seconds. Kindly let me know if I need to enable any configs specific to “neutron-sriov-agent” service. # tail -f /var/log/neutron/neutron-sriov-agent.log 2025-03-06 15:02:42.968 9311 INFO neutron.common.config [-] Logging enabled! 2025-03-06 15:02:42.969 9311 INFO neutron.common.config [-] /usr/bin/neutron-sriov-nic-agent version 24.0.0 2025-03-06 15:02:42.969 9311 DEBUG neutron.common.config [-] command line: /usr/bin/neutron-sriov-nic-agent --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/sriov_agent.ini --log-file=/var/log/neutron/neutron-sriov-agent.log setup_logging /usr/lib/python3/dist-packages/neutron/common/config.py:123 2025-03-06 15:02:42.970 9311 DEBUG neutron.agent.common.utils [-] Std library socket module is patched by eventlet. Requesting std library socket module from eventlet. __init__ /usr/lib/python3/dist-packages/neutron/agent/common/utils.py:91 2025-03-06 15:02:42.971 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Physical Devices mappings: {'vf_inside': ['enp216s0f0'], 'vf_outside': ['enp216s0f1']} 2025-03-06 15:02:42.971 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Exclude Devices: {} 2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider bandwidths: {} 2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider inventory defaults: {'allocation_ratio': 1.0, 'min_unit': 1, 'step_size': 1, 'reserved': 0} 2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider hypervisors: {'enp216s0f1': 'u36c01p06-ostest-cmp2.openstack.local', 'enp216s0f0': 'u36c01p06-ostest-cmp2.openstack.local'} 2025-03-06 15:02:42.973 9311 DEBUG neutron.plugins.ml2.drivers.mech_sriov.agent.eswitch_manager [-] Number of VFs configured on device enp216s0f0: 8 get_numvfs /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py:134 2025-03-06 15:02:42.973 9311 DEBUG neutron.plugins.ml2.drivers.mech_sriov.agent.eswitch_manager [-] Number of VFs configured on device enp216s0f1: 8 get_numvfs /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py:134 2025-03-06 15:02:42.976 9311 DEBUG neutron.agent.securitygroups_rpc [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Init firewall settings (driver=noop) init_firewall /usr/lib/python3/dist-packages/neutron/agent/securitygroups_rpc.py:124 2025-03-06 15:02:42.976 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] RPC agent_id: nic-switch-agent.u36c01p06-ostest-cmp2 2025-03-06 15:02:42.981 9311 INFO neutron.agent.agent_extensions_manager [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Loaded agent extensions: [] 2025-03-06 15:02:43.037 9311 WARNING oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true' 2025-03-06 15:02:43.038 9311 INFO oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] [7d2d8221-cd72-4203-b717-46c9d4fac7a9] Retrying to declare the exchange (q-agent-notifier-port-update_fanout) as non durable 2025-03-06 15:02:43.041 9311 ERROR oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Failed to declare consumer for topic 'q-agent-notifier-port-update': Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true' 2025-03-06 15:02:43.046 9311 ERROR oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Unable to connect to AMQP server on 10.10.4.19:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true' 2025-03-06 15:02:43.047 9311 CRITICAL neutron [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Unhandled error: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.10.4.19:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true' 2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last): 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 421, in declare 2025-03-06 15:02:43.047 9311 ERROR neutron self.queue.declare() 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 605, in declare 2025-03-06 15:02:43.047 9311 ERROR neutron self._create_exchange(nowait=nowait, channel=channel) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 612, in _create_exchange 2025-03-06 15:02:43.047 9311 ERROR neutron self.exchange.declare(nowait=nowait, channel=channel) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 180, in declare 2025-03-06 15:02:43.047 9311 ERROR neutron return (channel or self.channel).exchange_declare( 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 612, in exchange_declare 2025-03-06 15:02:43.047 9311 ERROR neutron self.send_method( 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 66, in send_method 2025-03-06 15:02:43.047 9311 ERROR neutron return self.wait(wait, returns_tuple=returns_tuple) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 86, in wait 2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.drain_events(timeout=timeout) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 522, in drain_events 2025-03-06 15:02:43.047 9311 ERROR neutron while not self.blocking_read(timeout): 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 528, in blocking_read 2025-03-06 15:02:43.047 9311 ERROR neutron return self.on_inbound_frame(frame) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/method_framing.py", line 53, in on_frame 2025-03-06 15:02:43.047 9311 ERROR neutron callback(channel, method_sig, buf, None) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 534, in on_inbound_method 2025-03-06 15:02:43.047 9311 ERROR neutron return self.channels[channel_id].dispatch_method( 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 143, in dispatch_method 2025-03-06 15:02:43.047 9311 ERROR neutron listener(*args) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 281, in _on_close 2025-03-06 15:02:43.047 9311 ERROR neutron raise error_for_code( 2025-03-06 15:02:43.047 9311 ERROR neutron amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true' 2025-03-06 15:02:43.047 9311 ERROR neutron 2025-03-06 15:02:43.047 9311 ERROR neutron During handling of the above exception, another exception occurred: 2025-03-06 15:02:43.047 9311 ERROR neutron 2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last): 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1036, in ensure 2025-03-06 15:02:43.047 9311 ERROR neutron ret, channel = autoretry_method() 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/connection.py", line 524, in _ensured 2025-03-06 15:02:43.047 9311 ERROR neutron return fun(*args, **kwargs) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/connection.py", line 600, in __call__ 2025-03-06 15:02:43.047 9311 ERROR neutron return fun(*args, channel=channels[0], **kwargs), channels[0] 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1025, in execute_method 2025-03-06 15:02:43.047 9311 ERROR neutron method() 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1262, in _declare_consumer 2025-03-06 15:02:43.047 9311 ERROR neutron consumer.declare(self) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 431, in declare 2025-03-06 15:02:43.047 9311 ERROR neutron self._declare_fallback(err, conn, consumer_arguments) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 396, in _declare_fallback 2025-03-06 15:02:43.047 9311 ERROR neutron self.queue.declare() 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 605, in declare 2025-03-06 15:02:43.047 9311 ERROR neutron self._create_exchange(nowait=nowait, channel=channel) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 612, in _create_exchange 2025-03-06 15:02:43.047 9311 ERROR neutron self.exchange.declare(nowait=nowait, channel=channel) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 180, in declare 2025-03-06 15:02:43.047 9311 ERROR neutron return (channel or self.channel).exchange_declare( 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 612, in exchange_declare 2025-03-06 15:02:43.047 9311 ERROR neutron self.send_method( 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 66, in send_method 2025-03-06 15:02:43.047 9311 ERROR neutron return self.wait(wait, returns_tuple=returns_tuple) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 86, in wait 2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.drain_events(timeout=timeout) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 522, in drain_events 2025-03-06 15:02:43.047 9311 ERROR neutron while not self.blocking_read(timeout): 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 528, in blocking_read 2025-03-06 15:02:43.047 9311 ERROR neutron return self.on_inbound_frame(frame) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/method_framing.py", line 53, in on_frame 2025-03-06 15:02:43.047 9311 ERROR neutron callback(channel, method_sig, buf, None) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 534, in on_inbound_method 2025-03-06 15:02:43.047 9311 ERROR neutron return self.channels[channel_id].dispatch_method( 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 143, in dispatch_method 2025-03-06 15:02:43.047 9311 ERROR neutron listener(*args) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 281, in _on_close 2025-03-06 15:02:43.047 9311 ERROR neutron raise error_for_code( 2025-03-06 15:02:43.047 9311 ERROR neutron amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true' 2025-03-06 15:02:43.047 9311 ERROR neutron 2025-03-06 15:02:43.047 9311 ERROR neutron During handling of the above exception, another exception occurred: 2025-03-06 15:02:43.047 9311 ERROR neutron 2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last): 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/bin/neutron-sriov-nic-agent", line 10, in <module> 2025-03-06 15:02:43.047 9311 ERROR neutron sys.exit(main()) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/cmd/eventlet/plugins/sriov_nic_neutron_agent.py", line 27, in main 2025-03-06 15:02:43.047 9311 ERROR neutron agent_main.main() 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 568, in main 2025-03-06 15:02:43.047 9311 ERROR neutron agent = SriovNicSwitchAgent(device_mappings, 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 181, in __init__ 2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.consume_in_threads() 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/rpc.py", line 345, in consume_in_threads 2025-03-06 15:02:43.047 9311 ERROR neutron server.start() 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 267, in wrapper 2025-03-06 15:02:43.047 9311 ERROR neutron states[state].run_once(lambda: fn(self, *args, **kwargs), 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 188, in run_once 2025-03-06 15:02:43.047 9311 ERROR neutron post_fn = fn() 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 267, in <lambda> 2025-03-06 15:02:43.047 9311 ERROR neutron states[state].run_once(lambda: fn(self, *args, **kwargs), 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 413, in start 2025-03-06 15:02:43.047 9311 ERROR neutron self.listener = self._create_listener() 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 150, in _create_listener 2025-03-06 15:02:43.047 9311 ERROR neutron return self.transport._listen(self._target, 1, None) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 142, in _listen 2025-03-06 15:02:43.047 9311 ERROR neutron return self._driver.listen(target, batch_size, 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 709, in listen 2025-03-06 15:02:43.047 9311 ERROR neutron conn.declare_fanout_consumer(target.topic, listener) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1398, in declare_fanout_consumer 2025-03-06 15:02:43.047 9311 ERROR neutron self.declare_consumer(consumer) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1273, in declare_consumer 2025-03-06 15:02:43.047 9311 ERROR neutron return self.ensure(_declare_consumer, 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1058, in ensure 2025-03-06 15:02:43.047 9311 ERROR neutron raise exceptions.MessageDeliveryFailure(msg) 2025-03-06 15:02:43.047 9311 ERROR neutron oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.10.4.19:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true' 2025-03-06 15:02:43.047 9311 ERROR neutron Thanks, Manohar