Hi Everyone,

I have the 2024.1 Caracal multi node setup working. Now I am trying to enable SRIOV on this setup by following the below link,

https://docs.openstack.org/neutron/2024.1/admin/config-sriov.html

Observing the below issue while running the “neutron-sriov-agent” service on compute node and it is restarting on every 2 seconds.

Kindly let me know if I need to enable any configs specific to “neutron-sriov-agent” service.

# tail -f /var/log/neutron/neutron-sriov-agent.log

2025-03-06 15:02:42.968 9311 INFO neutron.common.config [-] Logging enabled!

2025-03-06 15:02:42.969 9311 INFO neutron.common.config [-] /usr/bin/neutron-sriov-nic-agent version 24.0.0

2025-03-06 15:02:42.969 9311 DEBUG neutron.common.config [-] command line: /usr/bin/neutron-sriov-nic-agent --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/sriov_agent.ini --log-file=/var/log/neutron/neutron-sriov-agent.log setup_logging /usr/lib/python3/dist-packages/neutron/common/config.py:123

2025-03-06 15:02:42.970 9311 DEBUG neutron.agent.common.utils [-] Std library socket module is patched by eventlet. Requesting std library socket module from eventlet. __init__ /usr/lib/python3/dist-packages/neutron/agent/common/utils.py:91

2025-03-06 15:02:42.971 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Physical Devices mappings: {'vf_inside': ['enp216s0f0'], 'vf_outside': ['enp216s0f1']}

2025-03-06 15:02:42.971 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Exclude Devices: {}

2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider bandwidths: {}

2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider inventory defaults: {'allocation_ratio': 1.0, 'min_unit': 1, 'step_size': 1, 'reserved': 0}

2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider hypervisors: {'enp216s0f1': 'u36c01p06-ostest-cmp2.openstack.local', 'enp216s0f0': 'u36c01p06-ostest-cmp2.openstack.local'}

2025-03-06 15:02:42.973 9311 DEBUG neutron.plugins.ml2.drivers.mech_sriov.agent.eswitch_manager [-] Number of VFs configured on device enp216s0f0: 8 get_numvfs /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py:134

2025-03-06 15:02:42.973 9311 DEBUG neutron.plugins.ml2.drivers.mech_sriov.agent.eswitch_manager [-] Number of VFs configured on device enp216s0f1: 8 get_numvfs /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py:134

2025-03-06 15:02:42.976 9311 DEBUG neutron.agent.securitygroups_rpc [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Init firewall settings (driver=noop) init_firewall /usr/lib/python3/dist-packages/neutron/agent/securitygroups_rpc.py:124

2025-03-06 15:02:42.976 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] RPC agent_id: nic-switch-agent.u36c01p06-ostest-cmp2

2025-03-06 15:02:42.981 9311 INFO neutron.agent.agent_extensions_manager [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Loaded agent extensions: []

2025-03-06 15:02:43.037 9311 WARNING oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'

2025-03-06 15:02:43.038 9311 INFO oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] [7d2d8221-cd72-4203-b717-46c9d4fac7a9] Retrying to declare the exchange (q-agent-notifier-port-update_fanout) as non durable

2025-03-06 15:02:43.041 9311 ERROR oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Failed to declare consumer for topic 'q-agent-notifier-port-update': Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'

2025-03-06 15:02:43.046 9311 ERROR oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Unable to connect to AMQP server on 10.10.4.19:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'

2025-03-06 15:02:43.047 9311 CRITICAL neutron [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Unhandled error: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.10.4.19:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'

2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last):

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 421, in declare

2025-03-06 15:02:43.047 9311 ERROR neutron self.queue.declare()

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 605, in declare

2025-03-06 15:02:43.047 9311 ERROR neutron self._create_exchange(nowait=nowait, channel=channel)

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 612, in _create_exchange

2025-03-06 15:02:43.047 9311 ERROR neutron self.exchange.declare(nowait=nowait, channel=channel)

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 180, in declare

2025-03-06 15:02:43.047 9311 ERROR neutron return (channel or self.channel).exchange_declare(

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 612, in exchange_declare

2025-03-06 15:02:43.047 9311 ERROR neutron self.send_method(

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 66, in send_method

2025-03-06 15:02:43.047 9311 ERROR neutron return self.wait(wait, returns_tuple=returns_tuple)

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 86, in wait

2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.drain_events(timeout=timeout)

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 522, in drain_events

2025-03-06 15:02:43.047 9311 ERROR neutron while not self.blocking_read(timeout):

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 528, in blocking_read

2025-03-06 15:02:43.047 9311 ERROR neutron return self.on_inbound_frame(frame)

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/method_framing.py", line 53, in on_frame

2025-03-06 15:02:43.047 9311 ERROR neutron callback(channel, method_sig, buf, None)

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 534, in on_inbound_method

2025-03-06 15:02:43.047 9311 ERROR neutron return self.channels[channel_id].dispatch_method(

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 143, in dispatch_method

2025-03-06 15:02:43.047 9311 ERROR neutron listener(*args)

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 281, in _on_close

2025-03-06 15:02:43.047 9311 ERROR neutron raise error_for_code(

2025-03-06 15:02:43.047 9311 ERROR neutron amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'

2025-03-06 15:02:43.047 9311 ERROR neutron

2025-03-06 15:02:43.047 9311 ERROR neutron During handling of the above exception, another exception occurred:

2025-03-06 15:02:43.047 9311 ERROR neutron

2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last):

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1036, in ensure

2025-03-06 15:02:43.047 9311 ERROR neutron ret, channel = autoretry_method()

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/connection.py", line 524, in _ensured

2025-03-06 15:02:43.047 9311 ERROR neutron return fun(*args, **kwargs)

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/connection.py", line 600, in __call__

2025-03-06 15:02:43.047 9311 ERROR neutron return fun(*args, channel=channels[0], **kwargs), channels[0]

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1025, in execute_method

2025-03-06 15:02:43.047 9311 ERROR neutron method()

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1262, in _declare_consumer

2025-03-06 15:02:43.047 9311 ERROR neutron consumer.declare(self)

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 431, in declare

2025-03-06 15:02:43.047 9311 ERROR neutron self._declare_fallback(err, conn, consumer_arguments)

2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 396, in _declare_fallback