that not a sriov or neutron speicfc error. the sriov nic agent is talking to the neutron server over rabbit mq all client of the same vhost on rabbt need to consitetly have the same configuration for durable queue. you have a missmatch btween how the sriov agent is configured and the rest of neutron https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_driv... https://docs.openstack.org/neutron/latest/configuration/neutron.html#oslo-me... you just need to update the cofnig in the agent to match what you have on the controller and what is configured in your rabbit cluster. On 06/03/2025 09:42, Manohar Kundapura (makundap) wrote:
Hi Everyone,
I have the 2024.1 Caracal multi node setup working. Now I am trying to enable SRIOV on this setup by following the below link,
https://docs.openstack.org/neutron/2024.1/admin/config-sriov.html
Observing the below issue while running the “neutron-sriov-agent” service on compute node and it is restarting on every 2 seconds.
Kindly let me know if I need to enable any configs specific to “neutron-sriov-agent” service.
*/# tail -f /var/log/neutron/neutron-sriov-agent.log/*
/2025-03-06 15:02:42.968 9311 INFO neutron.common.config [-] Logging enabled!/
/2025-03-06 15:02:42.969 9311 INFO neutron.common.config [-] /usr/bin/neutron-sriov-nic-agent version 24.0.0/
/2025-03-06 15:02:42.969 9311 DEBUG neutron.common.config [-] command line: /usr/bin/neutron-sriov-nic-agent --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/sriov_agent.ini --log-file=/var/log/neutron/neutron-sriov-agent.log setup_logging /usr/lib/python3/dist-packages/neutron/common/config.py:123/
/2025-03-06 15:02:42.970 9311 DEBUG neutron.agent.common.utils [-] Std library socket module is patched by eventlet. Requesting std library socket module from eventlet. __init__ /usr/lib/python3/dist-packages/neutron/agent/common/utils.py:91/
/2025-03-06 15:02:42.971 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Physical Devices mappings: {'vf_inside': ['enp216s0f0'], 'vf_outside': ['enp216s0f1']}/
/2025-03-06 15:02:42.971 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Exclude Devices: {}/
/2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider bandwidths: {}/
/2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider inventory defaults: {'allocation_ratio': 1.0, 'min_unit': 1, 'step_size': 1, 'reserved': 0}/
/2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider hypervisors: {'enp216s0f1': 'u36c01p06-ostest-cmp2.openstack.local', 'enp216s0f0': 'u36c01p06-ostest-cmp2.openstack.local'}/
/2025-03-06 15:02:42.973 9311 DEBUG neutron.plugins.ml2.drivers.mech_sriov.agent.eswitch_manager [-] Number of VFs configured on device enp216s0f0: 8 get_numvfs /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py:134/
/2025-03-06 15:02:42.973 9311 DEBUG neutron.plugins.ml2.drivers.mech_sriov.agent.eswitch_manager [-] Number of VFs configured on device enp216s0f1: 8 get_numvfs /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py:134/
/2025-03-06 15:02:42.976 9311 DEBUG neutron.agent.securitygroups_rpc [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Init firewall settings (driver=noop) init_firewall /usr/lib/python3/dist-packages/neutron/agent/securitygroups_rpc.py:124/
/2025-03-06 15:02:42.976 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] RPC agent_id: nic-switch-agent.u36c01p06-ostest-cmp2/
/2025-03-06 15:02:42.981 9311 INFO neutron.agent.agent_extensions_manager [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Loaded agent extensions: []/
/2025-03-06 15:02:43.037 9311 WARNING oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'/
/2025-03-06 15:02:43.038 9311 INFO oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] [7d2d8221-cd72-4203-b717-46c9d4fac7a9] Retrying to declare the exchange (q-agent-notifier-port-update_fanout) as non durable/
/2025-03-06 15:02:43.041 9311 ERROR oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Failed to declare consumer for topic 'q-agent-notifier-port-update': Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'/
/2025-03-06 15:02:43.046 9311 ERROR oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Unable to connect to AMQP server on 10.10.4.19:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'/
/2025-03-06 15:02:43.047 9311 CRITICAL neutron [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Unhandled error: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.10.4.19:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'/
/2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last):/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 421, in declare/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.queue.declare()/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 605, in declare/
/2025-03-06 15:02:43.047 9311 ERROR neutron self._create_exchange(nowait=nowait, channel=channel)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 612, in _create_exchange/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.exchange.declare(nowait=nowait, channel=channel)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 180, in declare/
/2025-03-06 15:02:43.047 9311 ERROR neutron return (channel or self.channel).exchange_declare(/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 612, in exchange_declare/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.send_method(/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 66, in send_method/
/2025-03-06 15:02:43.047 9311 ERROR neutron return self.wait(wait, returns_tuple=returns_tuple)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 86, in wait/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.drain_events(timeout=timeout)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 522, in drain_events/
/2025-03-06 15:02:43.047 9311 ERROR neutron while not self.blocking_read(timeout):/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 528, in blocking_read/
/2025-03-06 15:02:43.047 9311 ERROR neutron return self.on_inbound_frame(frame)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/method_framing.py", line 53, in on_frame/
/2025-03-06 15:02:43.047 9311 ERROR neutron callback(channel, method_sig, buf, None)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 534, in on_inbound_method/
/2025-03-06 15:02:43.047 9311 ERROR neutron return self.channels[channel_id].dispatch_method(/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 143, in dispatch_method/
/2025-03-06 15:02:43.047 9311 ERROR neutron listener(*args)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 281, in _on_close/
/2025-03-06 15:02:43.047 9311 ERROR neutron raise error_for_code(/
/2025-03-06 15:02:43.047 9311 ERROR neutron amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'/
/2025-03-06 15:02:43.047 9311 ERROR neutron /
/2025-03-06 15:02:43.047 9311 ERROR neutron During handling of the above exception, another exception occurred:/
/2025-03-06 15:02:43.047 9311 ERROR neutron /
/2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last):/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1036, in ensure/
/2025-03-06 15:02:43.047 9311 ERROR neutron ret, channel = autoretry_method()/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/connection.py", line 524, in _ensured/
/2025-03-06 15:02:43.047 9311 ERROR neutron return fun(*args, **kwargs)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/connection.py", line 600, in __call__/
/2025-03-06 15:02:43.047 9311 ERROR neutron return fun(*args, channel=channels[0], **kwargs), channels[0]/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1025, in execute_method/
/2025-03-06 15:02:43.047 9311 ERROR neutron method()/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1262, in _declare_consumer/
/2025-03-06 15:02:43.047 9311 ERROR neutron consumer.declare(self)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 431, in declare/
/2025-03-06 15:02:43.047 9311 ERROR neutron self._declare_fallback(err, conn, consumer_arguments)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 396, in _declare_fallback/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.queue.declare()/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 605, in declare/
/2025-03-06 15:02:43.047 9311 ERROR neutron self._create_exchange(nowait=nowait, channel=channel)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 612, in _create_exchange/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.exchange.declare(nowait=nowait, channel=channel)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 180, in declare/
/2025-03-06 15:02:43.047 9311 ERROR neutron return (channel or self.channel).exchange_declare(/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 612, in exchange_declare/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.send_method(/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 66, in send_method/
/2025-03-06 15:02:43.047 9311 ERROR neutron return self.wait(wait, returns_tuple=returns_tuple)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 86, in wait/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.drain_events(timeout=timeout)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 522, in drain_events/
/2025-03-06 15:02:43.047 9311 ERROR neutron while not self.blocking_read(timeout):/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 528, in blocking_read/
/2025-03-06 15:02:43.047 9311 ERROR neutron return self.on_inbound_frame(frame)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/method_framing.py", line 53, in on_frame/
/2025-03-06 15:02:43.047 9311 ERROR neutron callback(channel, method_sig, buf, None)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 534, in on_inbound_method/
/2025-03-06 15:02:43.047 9311 ERROR neutron return self.channels[channel_id].dispatch_method(/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 143, in dispatch_method/
/2025-03-06 15:02:43.047 9311 ERROR neutron listener(*args)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 281, in _on_close/
/2025-03-06 15:02:43.047 9311 ERROR neutron raise error_for_code(/
/2025-03-06 15:02:43.047 9311 ERROR neutron amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'/
/2025-03-06 15:02:43.047 9311 ERROR neutron /
/2025-03-06 15:02:43.047 9311 ERROR neutron During handling of the above exception, another exception occurred:/
/2025-03-06 15:02:43.047 9311 ERROR neutron /
/2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last):/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/bin/neutron-sriov-nic-agent", line 10, in <module>/
/2025-03-06 15:02:43.047 9311 ERROR neutron sys.exit(main())/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/cmd/eventlet/plugins/sriov_nic_neutron_agent.py", line 27, in main/
/2025-03-06 15:02:43.047 9311 ERROR neutron agent_main.main()/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 568, in main/
/2025-03-06 15:02:43.047 9311 ERROR neutron agent = SriovNicSwitchAgent(device_mappings,/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 181, in __init__/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.consume_in_threads()/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/rpc.py", line 345, in consume_in_threads/
/2025-03-06 15:02:43.047 9311 ERROR neutron server.start()/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 267, in wrapper/
/2025-03-06 15:02:43.047 9311 ERROR neutron states[state].run_once(lambda: fn(self, *args, **kwargs),/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 188, in run_once/
/2025-03-06 15:02:43.047 9311 ERROR neutron post_fn = fn()/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 267, in <lambda>/
/2025-03-06 15:02:43.047 9311 ERROR neutron states[state].run_once(lambda: fn(self, *args, **kwargs),/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 413, in start/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.listener = self._create_listener()/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 150, in _create_listener/
/2025-03-06 15:02:43.047 9311 ERROR neutron return self.transport._listen(self._target, 1, None)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 142, in _listen/
/2025-03-06 15:02:43.047 9311 ERROR neutron return self._driver.listen(target, batch_size,/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 709, in listen/
/2025-03-06 15:02:43.047 9311 ERROR neutron conn.declare_fanout_consumer(target.topic, listener)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1398, in declare_fanout_consumer/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.declare_consumer(consumer)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1273, in declare_consumer/
/2025-03-06 15:02:43.047 9311 ERROR neutron return self.ensure(_declare_consumer,/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1058, in ensure/
/2025-03-06 15:02:43.047 9311 ERROR neutron raise exceptions.MessageDeliveryFailure(msg)/
/2025-03-06 15:02:43.047 9311 ERROR neutron oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.10.4.19:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'/
/2025-03-06 15:02:43.047 9311 ERROR neutron/
Thanks,
Manohar//