SRIOV on 2024.1 caracal
Hi Everyone, I have the 2024.1 Caracal multi node setup working. Now I am trying to enable SRIOV on this setup by following the below link, https://docs.openstack.org/neutron/2024.1/admin/config-sriov.html Observing the below issue while running the “neutron-sriov-agent” service on compute node and it is restarting on every 2 seconds. Kindly let me know if I need to enable any configs specific to “neutron-sriov-agent” service. # tail -f /var/log/neutron/neutron-sriov-agent.log 2025-03-06 15:02:42.968 9311 INFO neutron.common.config [-] Logging enabled! 2025-03-06 15:02:42.969 9311 INFO neutron.common.config [-] /usr/bin/neutron-sriov-nic-agent version 24.0.0 2025-03-06 15:02:42.969 9311 DEBUG neutron.common.config [-] command line: /usr/bin/neutron-sriov-nic-agent --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/sriov_agent.ini --log-file=/var/log/neutron/neutron-sriov-agent.log setup_logging /usr/lib/python3/dist-packages/neutron/common/config.py:123 2025-03-06 15:02:42.970 9311 DEBUG neutron.agent.common.utils [-] Std library socket module is patched by eventlet. Requesting std library socket module from eventlet. __init__ /usr/lib/python3/dist-packages/neutron/agent/common/utils.py:91 2025-03-06 15:02:42.971 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Physical Devices mappings: {'vf_inside': ['enp216s0f0'], 'vf_outside': ['enp216s0f1']} 2025-03-06 15:02:42.971 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Exclude Devices: {} 2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider bandwidths: {} 2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider inventory defaults: {'allocation_ratio': 1.0, 'min_unit': 1, 'step_size': 1, 'reserved': 0} 2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider hypervisors: {'enp216s0f1': 'u36c01p06-ostest-cmp2.openstack.local', 'enp216s0f0': 'u36c01p06-ostest-cmp2.openstack.local'} 2025-03-06 15:02:42.973 9311 DEBUG neutron.plugins.ml2.drivers.mech_sriov.agent.eswitch_manager [-] Number of VFs configured on device enp216s0f0: 8 get_numvfs /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py:134 2025-03-06 15:02:42.973 9311 DEBUG neutron.plugins.ml2.drivers.mech_sriov.agent.eswitch_manager [-] Number of VFs configured on device enp216s0f1: 8 get_numvfs /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py:134 2025-03-06 15:02:42.976 9311 DEBUG neutron.agent.securitygroups_rpc [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Init firewall settings (driver=noop) init_firewall /usr/lib/python3/dist-packages/neutron/agent/securitygroups_rpc.py:124 2025-03-06 15:02:42.976 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] RPC agent_id: nic-switch-agent.u36c01p06-ostest-cmp2 2025-03-06 15:02:42.981 9311 INFO neutron.agent.agent_extensions_manager [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Loaded agent extensions: [] 2025-03-06 15:02:43.037 9311 WARNING oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true' 2025-03-06 15:02:43.038 9311 INFO oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] [7d2d8221-cd72-4203-b717-46c9d4fac7a9] Retrying to declare the exchange (q-agent-notifier-port-update_fanout) as non durable 2025-03-06 15:02:43.041 9311 ERROR oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Failed to declare consumer for topic 'q-agent-notifier-port-update': Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true' 2025-03-06 15:02:43.046 9311 ERROR oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Unable to connect to AMQP server on 10.10.4.19:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true' 2025-03-06 15:02:43.047 9311 CRITICAL neutron [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Unhandled error: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.10.4.19:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true' 2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last): 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 421, in declare 2025-03-06 15:02:43.047 9311 ERROR neutron self.queue.declare() 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 605, in declare 2025-03-06 15:02:43.047 9311 ERROR neutron self._create_exchange(nowait=nowait, channel=channel) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 612, in _create_exchange 2025-03-06 15:02:43.047 9311 ERROR neutron self.exchange.declare(nowait=nowait, channel=channel) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 180, in declare 2025-03-06 15:02:43.047 9311 ERROR neutron return (channel or self.channel).exchange_declare( 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 612, in exchange_declare 2025-03-06 15:02:43.047 9311 ERROR neutron self.send_method( 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 66, in send_method 2025-03-06 15:02:43.047 9311 ERROR neutron return self.wait(wait, returns_tuple=returns_tuple) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 86, in wait 2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.drain_events(timeout=timeout) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 522, in drain_events 2025-03-06 15:02:43.047 9311 ERROR neutron while not self.blocking_read(timeout): 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 528, in blocking_read 2025-03-06 15:02:43.047 9311 ERROR neutron return self.on_inbound_frame(frame) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/method_framing.py", line 53, in on_frame 2025-03-06 15:02:43.047 9311 ERROR neutron callback(channel, method_sig, buf, None) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 534, in on_inbound_method 2025-03-06 15:02:43.047 9311 ERROR neutron return self.channels[channel_id].dispatch_method( 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 143, in dispatch_method 2025-03-06 15:02:43.047 9311 ERROR neutron listener(*args) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 281, in _on_close 2025-03-06 15:02:43.047 9311 ERROR neutron raise error_for_code( 2025-03-06 15:02:43.047 9311 ERROR neutron amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true' 2025-03-06 15:02:43.047 9311 ERROR neutron 2025-03-06 15:02:43.047 9311 ERROR neutron During handling of the above exception, another exception occurred: 2025-03-06 15:02:43.047 9311 ERROR neutron 2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last): 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1036, in ensure 2025-03-06 15:02:43.047 9311 ERROR neutron ret, channel = autoretry_method() 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/connection.py", line 524, in _ensured 2025-03-06 15:02:43.047 9311 ERROR neutron return fun(*args, **kwargs) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/connection.py", line 600, in __call__ 2025-03-06 15:02:43.047 9311 ERROR neutron return fun(*args, channel=channels[0], **kwargs), channels[0] 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1025, in execute_method 2025-03-06 15:02:43.047 9311 ERROR neutron method() 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1262, in _declare_consumer 2025-03-06 15:02:43.047 9311 ERROR neutron consumer.declare(self) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 431, in declare 2025-03-06 15:02:43.047 9311 ERROR neutron self._declare_fallback(err, conn, consumer_arguments) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 396, in _declare_fallback 2025-03-06 15:02:43.047 9311 ERROR neutron self.queue.declare() 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 605, in declare 2025-03-06 15:02:43.047 9311 ERROR neutron self._create_exchange(nowait=nowait, channel=channel) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 612, in _create_exchange 2025-03-06 15:02:43.047 9311 ERROR neutron self.exchange.declare(nowait=nowait, channel=channel) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 180, in declare 2025-03-06 15:02:43.047 9311 ERROR neutron return (channel or self.channel).exchange_declare( 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 612, in exchange_declare 2025-03-06 15:02:43.047 9311 ERROR neutron self.send_method( 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 66, in send_method 2025-03-06 15:02:43.047 9311 ERROR neutron return self.wait(wait, returns_tuple=returns_tuple) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 86, in wait 2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.drain_events(timeout=timeout) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 522, in drain_events 2025-03-06 15:02:43.047 9311 ERROR neutron while not self.blocking_read(timeout): 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 528, in blocking_read 2025-03-06 15:02:43.047 9311 ERROR neutron return self.on_inbound_frame(frame) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/method_framing.py", line 53, in on_frame 2025-03-06 15:02:43.047 9311 ERROR neutron callback(channel, method_sig, buf, None) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 534, in on_inbound_method 2025-03-06 15:02:43.047 9311 ERROR neutron return self.channels[channel_id].dispatch_method( 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 143, in dispatch_method 2025-03-06 15:02:43.047 9311 ERROR neutron listener(*args) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 281, in _on_close 2025-03-06 15:02:43.047 9311 ERROR neutron raise error_for_code( 2025-03-06 15:02:43.047 9311 ERROR neutron amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true' 2025-03-06 15:02:43.047 9311 ERROR neutron 2025-03-06 15:02:43.047 9311 ERROR neutron During handling of the above exception, another exception occurred: 2025-03-06 15:02:43.047 9311 ERROR neutron 2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last): 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/bin/neutron-sriov-nic-agent", line 10, in <module> 2025-03-06 15:02:43.047 9311 ERROR neutron sys.exit(main()) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/cmd/eventlet/plugins/sriov_nic_neutron_agent.py", line 27, in main 2025-03-06 15:02:43.047 9311 ERROR neutron agent_main.main() 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 568, in main 2025-03-06 15:02:43.047 9311 ERROR neutron agent = SriovNicSwitchAgent(device_mappings, 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 181, in __init__ 2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.consume_in_threads() 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/rpc.py", line 345, in consume_in_threads 2025-03-06 15:02:43.047 9311 ERROR neutron server.start() 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 267, in wrapper 2025-03-06 15:02:43.047 9311 ERROR neutron states[state].run_once(lambda: fn(self, *args, **kwargs), 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 188, in run_once 2025-03-06 15:02:43.047 9311 ERROR neutron post_fn = fn() 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 267, in <lambda> 2025-03-06 15:02:43.047 9311 ERROR neutron states[state].run_once(lambda: fn(self, *args, **kwargs), 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 413, in start 2025-03-06 15:02:43.047 9311 ERROR neutron self.listener = self._create_listener() 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 150, in _create_listener 2025-03-06 15:02:43.047 9311 ERROR neutron return self.transport._listen(self._target, 1, None) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 142, in _listen 2025-03-06 15:02:43.047 9311 ERROR neutron return self._driver.listen(target, batch_size, 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 709, in listen 2025-03-06 15:02:43.047 9311 ERROR neutron conn.declare_fanout_consumer(target.topic, listener) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1398, in declare_fanout_consumer 2025-03-06 15:02:43.047 9311 ERROR neutron self.declare_consumer(consumer) 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1273, in declare_consumer 2025-03-06 15:02:43.047 9311 ERROR neutron return self.ensure(_declare_consumer, 2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1058, in ensure 2025-03-06 15:02:43.047 9311 ERROR neutron raise exceptions.MessageDeliveryFailure(msg) 2025-03-06 15:02:43.047 9311 ERROR neutron oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.10.4.19:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true' 2025-03-06 15:02:43.047 9311 ERROR neutron Thanks, Manohar
Hi, To tell the truth I never seen such issue, but one hint what to check. The amqp_durable_queues setting is coming from oslo.messaging (see [1]) by default it is false, so one thing should be to check if all your services has this default, or accidentally one of them set it to true. You can see in the logs these values when the services are started, if they are not directly set in your Neutron config files. [1]: https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messagin... Best wishes Lajos Katona Manohar Kundapura (makundap) <makundap@cisco.com> ezt írta (időpont: 2025. márc. 6., Cs, 17:30):
Hi Everyone,
I have the 2024.1 Caracal multi node setup working. Now I am trying to enable SRIOV on this setup by following the below link,
https://docs.openstack.org/neutron/2024.1/admin/config-sriov.html
Observing the below issue while running the “neutron-sriov-agent” service on compute node and it is restarting on every 2 seconds.
Kindly let me know if I need to enable any configs specific to “neutron-sriov-agent” service.
*# tail -f /var/log/neutron/neutron-sriov-agent.log*
*2025-03-06 15:02:42.968 9311 INFO neutron.common.config [-] Logging enabled!*
*2025-03-06 15:02:42.969 9311 INFO neutron.common.config [-] /usr/bin/neutron-sriov-nic-agent version 24.0.0*
*2025-03-06 15:02:42.969 9311 DEBUG neutron.common.config [-] command line: /usr/bin/neutron-sriov-nic-agent --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/sriov_agent.ini --log-file=/var/log/neutron/neutron-sriov-agent.log setup_logging /usr/lib/python3/dist-packages/neutron/common/config.py:123*
*2025-03-06 15:02:42.970 9311 DEBUG neutron.agent.common.utils [-] Std library socket module is patched by eventlet. Requesting std library socket module from eventlet. __init__ /usr/lib/python3/dist-packages/neutron/agent/common/utils.py:91*
*2025-03-06 15:02:42.971 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Physical Devices mappings: {'vf_inside': ['enp216s0f0'], 'vf_outside': ['enp216s0f1']}*
*2025-03-06 15:02:42.971 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Exclude Devices: {}*
*2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider bandwidths: {}*
*2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider inventory defaults: {'allocation_ratio': 1.0, 'min_unit': 1, 'step_size': 1, 'reserved': 0}*
*2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider hypervisors: {'enp216s0f1': 'u36c01p06-ostest-cmp2.openstack.local', 'enp216s0f0': 'u36c01p06-ostest-cmp2.openstack.local'}*
*2025-03-06 15:02:42.973 9311 DEBUG neutron.plugins.ml2.drivers.mech_sriov.agent.eswitch_manager [-] Number of VFs configured on device enp216s0f0: 8 get_numvfs /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py:134*
*2025-03-06 15:02:42.973 9311 DEBUG neutron.plugins.ml2.drivers.mech_sriov.agent.eswitch_manager [-] Number of VFs configured on device enp216s0f1: 8 get_numvfs /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py:134*
*2025-03-06 15:02:42.976 9311 DEBUG neutron.agent.securitygroups_rpc [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Init firewall settings (driver=noop) init_firewall /usr/lib/python3/dist-packages/neutron/agent/securitygroups_rpc.py:124*
*2025-03-06 15:02:42.976 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] RPC agent_id: nic-switch-agent.u36c01p06-ostest-cmp2*
*2025-03-06 15:02:42.981 9311 INFO neutron.agent.agent_extensions_manager [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Loaded agent extensions: []*
*2025-03-06 15:02:43.037 9311 WARNING oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'*
*2025-03-06 15:02:43.038 9311 INFO oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] [7d2d8221-cd72-4203-b717-46c9d4fac7a9] Retrying to declare the exchange (q-agent-notifier-port-update_fanout) as non durable*
*2025-03-06 15:02:43.041 9311 ERROR oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Failed to declare consumer for topic 'q-agent-notifier-port-update': Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'*
*2025-03-06 15:02:43.046 9311 ERROR oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Unable to connect to AMQP server on 10.10.4.19:5671 <http://10.10.4.19:5671> after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'*
*2025-03-06 15:02:43.047 9311 CRITICAL neutron [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Unhandled error: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.10.4.19:5671 <http://10.10.4.19:5671> after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'*
*2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last):*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 421, in declare*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.queue.declare()*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 605, in declare*
*2025-03-06 15:02:43.047 9311 ERROR neutron self._create_exchange(nowait=nowait, channel=channel)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 612, in _create_exchange*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.exchange.declare(nowait=nowait, channel=channel)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 180, in declare*
*2025-03-06 15:02:43.047 9311 ERROR neutron return (channel or self.channel).exchange_declare(*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 612, in exchange_declare*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.send_method(*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 66, in send_method*
*2025-03-06 15:02:43.047 9311 ERROR neutron return self.wait(wait, returns_tuple=returns_tuple)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 86, in wait*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.drain_events(timeout=timeout)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 522, in drain_events*
*2025-03-06 15:02:43.047 9311 ERROR neutron while not self.blocking_read(timeout):*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 528, in blocking_read*
*2025-03-06 15:02:43.047 9311 ERROR neutron return self.on_inbound_frame(frame)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/method_framing.py", line 53, in on_frame*
*2025-03-06 15:02:43.047 9311 ERROR neutron callback(channel, method_sig, buf, None)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 534, in on_inbound_method*
*2025-03-06 15:02:43.047 9311 ERROR neutron return self.channels[channel_id].dispatch_method(*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 143, in dispatch_method*
*2025-03-06 15:02:43.047 9311 ERROR neutron listener(*args)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 281, in _on_close*
*2025-03-06 15:02:43.047 9311 ERROR neutron raise error_for_code(*
*2025-03-06 15:02:43.047 9311 ERROR neutron amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'*
*2025-03-06 15:02:43.047 9311 ERROR neutron *
*2025-03-06 15:02:43.047 9311 ERROR neutron During handling of the above exception, another exception occurred:*
*2025-03-06 15:02:43.047 9311 ERROR neutron *
*2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last):*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1036, in ensure*
*2025-03-06 15:02:43.047 9311 ERROR neutron ret, channel = autoretry_method()*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/connection.py", line 524, in _ensured*
*2025-03-06 15:02:43.047 9311 ERROR neutron return fun(*args, **kwargs)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/connection.py", line 600, in __call__*
*2025-03-06 15:02:43.047 9311 ERROR neutron return fun(*args, channel=channels[0], **kwargs), channels[0]*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1025, in execute_method*
*2025-03-06 15:02:43.047 9311 ERROR neutron method()*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1262, in _declare_consumer*
*2025-03-06 15:02:43.047 9311 ERROR neutron consumer.declare(self)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 431, in declare*
*2025-03-06 15:02:43.047 9311 ERROR neutron self._declare_fallback(err, conn, consumer_arguments)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 396, in _declare_fallback*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.queue.declare()*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 605, in declare*
*2025-03-06 15:02:43.047 9311 ERROR neutron self._create_exchange(nowait=nowait, channel=channel)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 612, in _create_exchange*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.exchange.declare(nowait=nowait, channel=channel)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 180, in declare*
*2025-03-06 15:02:43.047 9311 ERROR neutron return (channel or self.channel).exchange_declare(*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 612, in exchange_declare*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.send_method(*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 66, in send_method*
*2025-03-06 15:02:43.047 9311 ERROR neutron return self.wait(wait, returns_tuple=returns_tuple)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 86, in wait*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.drain_events(timeout=timeout)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 522, in drain_events*
*2025-03-06 15:02:43.047 9311 ERROR neutron while not self.blocking_read(timeout):*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 528, in blocking_read*
*2025-03-06 15:02:43.047 9311 ERROR neutron return self.on_inbound_frame(frame)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/method_framing.py", line 53, in on_frame*
*2025-03-06 15:02:43.047 9311 ERROR neutron callback(channel, method_sig, buf, None)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 534, in on_inbound_method*
*2025-03-06 15:02:43.047 9311 ERROR neutron return self.channels[channel_id].dispatch_method(*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 143, in dispatch_method*
*2025-03-06 15:02:43.047 9311 ERROR neutron listener(*args)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 281, in _on_close*
*2025-03-06 15:02:43.047 9311 ERROR neutron raise error_for_code(*
*2025-03-06 15:02:43.047 9311 ERROR neutron amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'*
*2025-03-06 15:02:43.047 9311 ERROR neutron *
*2025-03-06 15:02:43.047 9311 ERROR neutron During handling of the above exception, another exception occurred:*
*2025-03-06 15:02:43.047 9311 ERROR neutron *
*2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last):*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/bin/neutron-sriov-nic-agent", line 10, in <module>*
*2025-03-06 15:02:43.047 9311 ERROR neutron sys.exit(main())*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/cmd/eventlet/plugins/sriov_nic_neutron_agent.py", line 27, in main*
*2025-03-06 15:02:43.047 9311 ERROR neutron agent_main.main()*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 568, in main*
*2025-03-06 15:02:43.047 9311 ERROR neutron agent = SriovNicSwitchAgent(device_mappings,*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 181, in __init__*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.consume_in_threads()*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/rpc.py", line 345, in consume_in_threads*
*2025-03-06 15:02:43.047 9311 ERROR neutron server.start()*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 267, in wrapper*
*2025-03-06 15:02:43.047 9311 ERROR neutron states[state].run_once(lambda: fn(self, *args, **kwargs),*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 188, in run_once*
*2025-03-06 15:02:43.047 9311 ERROR neutron post_fn = fn()*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 267, in <lambda>*
*2025-03-06 15:02:43.047 9311 ERROR neutron states[state].run_once(lambda: fn(self, *args, **kwargs),*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 413, in start*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.listener = self._create_listener()*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 150, in _create_listener*
*2025-03-06 15:02:43.047 9311 ERROR neutron return self.transport._listen(self._target, 1, None)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 142, in _listen*
*2025-03-06 15:02:43.047 9311 ERROR neutron return self._driver.listen(target, batch_size,*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 709, in listen*
*2025-03-06 15:02:43.047 9311 ERROR neutron conn.declare_fanout_consumer(target.topic, listener)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1398, in declare_fanout_consumer*
*2025-03-06 15:02:43.047 9311 ERROR neutron self.declare_consumer(consumer)*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1273, in declare_consumer*
*2025-03-06 15:02:43.047 9311 ERROR neutron return self.ensure(_declare_consumer,*
*2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1058, in ensure*
*2025-03-06 15:02:43.047 9311 ERROR neutron raise exceptions.MessageDeliveryFailure(msg)*
*2025-03-06 15:02:43.047 9311 ERROR neutron oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.10.4.19:5671 <http://10.10.4.19:5671> after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'*
*2025-03-06 15:02:43.047 9311 ERROR neutron*
Thanks,
Manohar
that not a sriov or neutron speicfc error. the sriov nic agent is talking to the neutron server over rabbit mq all client of the same vhost on rabbt need to consitetly have the same configuration for durable queue. you have a missmatch btween how the sriov agent is configured and the rest of neutron https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_driv... https://docs.openstack.org/neutron/latest/configuration/neutron.html#oslo-me... you just need to update the cofnig in the agent to match what you have on the controller and what is configured in your rabbit cluster. On 06/03/2025 09:42, Manohar Kundapura (makundap) wrote:
Hi Everyone,
I have the 2024.1 Caracal multi node setup working. Now I am trying to enable SRIOV on this setup by following the below link,
https://docs.openstack.org/neutron/2024.1/admin/config-sriov.html
Observing the below issue while running the “neutron-sriov-agent” service on compute node and it is restarting on every 2 seconds.
Kindly let me know if I need to enable any configs specific to “neutron-sriov-agent” service.
*/# tail -f /var/log/neutron/neutron-sriov-agent.log/*
/2025-03-06 15:02:42.968 9311 INFO neutron.common.config [-] Logging enabled!/
/2025-03-06 15:02:42.969 9311 INFO neutron.common.config [-] /usr/bin/neutron-sriov-nic-agent version 24.0.0/
/2025-03-06 15:02:42.969 9311 DEBUG neutron.common.config [-] command line: /usr/bin/neutron-sriov-nic-agent --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/sriov_agent.ini --log-file=/var/log/neutron/neutron-sriov-agent.log setup_logging /usr/lib/python3/dist-packages/neutron/common/config.py:123/
/2025-03-06 15:02:42.970 9311 DEBUG neutron.agent.common.utils [-] Std library socket module is patched by eventlet. Requesting std library socket module from eventlet. __init__ /usr/lib/python3/dist-packages/neutron/agent/common/utils.py:91/
/2025-03-06 15:02:42.971 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Physical Devices mappings: {'vf_inside': ['enp216s0f0'], 'vf_outside': ['enp216s0f1']}/
/2025-03-06 15:02:42.971 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Exclude Devices: {}/
/2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider bandwidths: {}/
/2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider inventory defaults: {'allocation_ratio': 1.0, 'min_unit': 1, 'step_size': 1, 'reserved': 0}/
/2025-03-06 15:02:42.972 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Resource provider hypervisors: {'enp216s0f1': 'u36c01p06-ostest-cmp2.openstack.local', 'enp216s0f0': 'u36c01p06-ostest-cmp2.openstack.local'}/
/2025-03-06 15:02:42.973 9311 DEBUG neutron.plugins.ml2.drivers.mech_sriov.agent.eswitch_manager [-] Number of VFs configured on device enp216s0f0: 8 get_numvfs /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py:134/
/2025-03-06 15:02:42.973 9311 DEBUG neutron.plugins.ml2.drivers.mech_sriov.agent.eswitch_manager [-] Number of VFs configured on device enp216s0f1: 8 get_numvfs /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py:134/
/2025-03-06 15:02:42.976 9311 DEBUG neutron.agent.securitygroups_rpc [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Init firewall settings (driver=noop) init_firewall /usr/lib/python3/dist-packages/neutron/agent/securitygroups_rpc.py:124/
/2025-03-06 15:02:42.976 9311 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] RPC agent_id: nic-switch-agent.u36c01p06-ostest-cmp2/
/2025-03-06 15:02:42.981 9311 INFO neutron.agent.agent_extensions_manager [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Loaded agent extensions: []/
/2025-03-06 15:02:43.037 9311 WARNING oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'/
/2025-03-06 15:02:43.038 9311 INFO oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] [7d2d8221-cd72-4203-b717-46c9d4fac7a9] Retrying to declare the exchange (q-agent-notifier-port-update_fanout) as non durable/
/2025-03-06 15:02:43.041 9311 ERROR oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Failed to declare consumer for topic 'q-agent-notifier-port-update': Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'/
/2025-03-06 15:02:43.046 9311 ERROR oslo.messaging._drivers.impl_rabbit [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Unable to connect to AMQP server on 10.10.4.19:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'/
/2025-03-06 15:02:43.047 9311 CRITICAL neutron [None req-617911dd-c075-4154-967b-a8ad1c334d34 - - - - - -] Unhandled error: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.10.4.19:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'/
/2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last):/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 421, in declare/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.queue.declare()/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 605, in declare/
/2025-03-06 15:02:43.047 9311 ERROR neutron self._create_exchange(nowait=nowait, channel=channel)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 612, in _create_exchange/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.exchange.declare(nowait=nowait, channel=channel)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 180, in declare/
/2025-03-06 15:02:43.047 9311 ERROR neutron return (channel or self.channel).exchange_declare(/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 612, in exchange_declare/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.send_method(/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 66, in send_method/
/2025-03-06 15:02:43.047 9311 ERROR neutron return self.wait(wait, returns_tuple=returns_tuple)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 86, in wait/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.drain_events(timeout=timeout)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 522, in drain_events/
/2025-03-06 15:02:43.047 9311 ERROR neutron while not self.blocking_read(timeout):/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 528, in blocking_read/
/2025-03-06 15:02:43.047 9311 ERROR neutron return self.on_inbound_frame(frame)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/method_framing.py", line 53, in on_frame/
/2025-03-06 15:02:43.047 9311 ERROR neutron callback(channel, method_sig, buf, None)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 534, in on_inbound_method/
/2025-03-06 15:02:43.047 9311 ERROR neutron return self.channels[channel_id].dispatch_method(/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 143, in dispatch_method/
/2025-03-06 15:02:43.047 9311 ERROR neutron listener(*args)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 281, in _on_close/
/2025-03-06 15:02:43.047 9311 ERROR neutron raise error_for_code(/
/2025-03-06 15:02:43.047 9311 ERROR neutron amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'/
/2025-03-06 15:02:43.047 9311 ERROR neutron /
/2025-03-06 15:02:43.047 9311 ERROR neutron During handling of the above exception, another exception occurred:/
/2025-03-06 15:02:43.047 9311 ERROR neutron /
/2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last):/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1036, in ensure/
/2025-03-06 15:02:43.047 9311 ERROR neutron ret, channel = autoretry_method()/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/connection.py", line 524, in _ensured/
/2025-03-06 15:02:43.047 9311 ERROR neutron return fun(*args, **kwargs)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/connection.py", line 600, in __call__/
/2025-03-06 15:02:43.047 9311 ERROR neutron return fun(*args, channel=channels[0], **kwargs), channels[0]/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1025, in execute_method/
/2025-03-06 15:02:43.047 9311 ERROR neutron method()/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1262, in _declare_consumer/
/2025-03-06 15:02:43.047 9311 ERROR neutron consumer.declare(self)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 431, in declare/
/2025-03-06 15:02:43.047 9311 ERROR neutron self._declare_fallback(err, conn, consumer_arguments)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 396, in _declare_fallback/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.queue.declare()/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 605, in declare/
/2025-03-06 15:02:43.047 9311 ERROR neutron self._create_exchange(nowait=nowait, channel=channel)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 612, in _create_exchange/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.exchange.declare(nowait=nowait, channel=channel)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/kombu/entity.py", line 180, in declare/
/2025-03-06 15:02:43.047 9311 ERROR neutron return (channel or self.channel).exchange_declare(/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 612, in exchange_declare/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.send_method(/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 66, in send_method/
/2025-03-06 15:02:43.047 9311 ERROR neutron return self.wait(wait, returns_tuple=returns_tuple)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 86, in wait/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.drain_events(timeout=timeout)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 522, in drain_events/
/2025-03-06 15:02:43.047 9311 ERROR neutron while not self.blocking_read(timeout):/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 528, in blocking_read/
/2025-03-06 15:02:43.047 9311 ERROR neutron return self.on_inbound_frame(frame)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/method_framing.py", line 53, in on_frame/
/2025-03-06 15:02:43.047 9311 ERROR neutron callback(channel, method_sig, buf, None)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/connection.py", line 534, in on_inbound_method/
/2025-03-06 15:02:43.047 9311 ERROR neutron return self.channels[channel_id].dispatch_method(/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 143, in dispatch_method/
/2025-03-06 15:02:43.047 9311 ERROR neutron listener(*args)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/amqp/channel.py", line 281, in _on_close/
/2025-03-06 15:02:43.047 9311 ERROR neutron raise error_for_code(/
/2025-03-06 15:02:43.047 9311 ERROR neutron amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'/
/2025-03-06 15:02:43.047 9311 ERROR neutron /
/2025-03-06 15:02:43.047 9311 ERROR neutron During handling of the above exception, another exception occurred:/
/2025-03-06 15:02:43.047 9311 ERROR neutron /
/2025-03-06 15:02:43.047 9311 ERROR neutron Traceback (most recent call last):/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/bin/neutron-sriov-nic-agent", line 10, in <module>/
/2025-03-06 15:02:43.047 9311 ERROR neutron sys.exit(main())/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/cmd/eventlet/plugins/sriov_nic_neutron_agent.py", line 27, in main/
/2025-03-06 15:02:43.047 9311 ERROR neutron agent_main.main()/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 568, in main/
/2025-03-06 15:02:43.047 9311 ERROR neutron agent = SriovNicSwitchAgent(device_mappings,/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 181, in __init__/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.connection.consume_in_threads()/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/rpc.py", line 345, in consume_in_threads/
/2025-03-06 15:02:43.047 9311 ERROR neutron server.start()/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 267, in wrapper/
/2025-03-06 15:02:43.047 9311 ERROR neutron states[state].run_once(lambda: fn(self, *args, **kwargs),/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 188, in run_once/
/2025-03-06 15:02:43.047 9311 ERROR neutron post_fn = fn()/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 267, in <lambda>/
/2025-03-06 15:02:43.047 9311 ERROR neutron states[state].run_once(lambda: fn(self, *args, **kwargs),/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 413, in start/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.listener = self._create_listener()/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 150, in _create_listener/
/2025-03-06 15:02:43.047 9311 ERROR neutron return self.transport._listen(self._target, 1, None)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 142, in _listen/
/2025-03-06 15:02:43.047 9311 ERROR neutron return self._driver.listen(target, batch_size,/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 709, in listen/
/2025-03-06 15:02:43.047 9311 ERROR neutron conn.declare_fanout_consumer(target.topic, listener)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1398, in declare_fanout_consumer/
/2025-03-06 15:02:43.047 9311 ERROR neutron self.declare_consumer(consumer)/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1273, in declare_consumer/
/2025-03-06 15:02:43.047 9311 ERROR neutron return self.ensure(_declare_consumer,/
/2025-03-06 15:02:43.047 9311 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1058, in ensure/
/2025-03-06 15:02:43.047 9311 ERROR neutron raise exceptions.MessageDeliveryFailure(msg)/
/2025-03-06 15:02:43.047 9311 ERROR neutron oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.10.4.19:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'q-agent-notifier-port-update_fanout' in vhost 'neutron': received 'false' but current is 'true'/
/2025-03-06 15:02:43.047 9311 ERROR neutron/
Thanks,
Manohar//
participants (3)
-
Lajos Katona
-
Manohar Kundapura (makundap)
-
Sean Mooney