It sounds like you've got an issue with your RabbitMQ infrastructure that's causing messages not to show up. I suggest focusing your troubleshooting there. I've seen these issues resolved by simply rebuilding the RabbitMQ cluster.
On Thu, Nov 11, 2021 at 6:07 PM DHilsbos@performair.com wrote:
All;
I'm running into a strange issue when I try to migrate a server from one host to another.
The only messages I'm seeing are in nova-conductor.log: 2021-11-11 15:16:49.118 1887742 WARNING nova.scheduler.utils [req-d648901b-d000-47ec-9497-bcc51586c381 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Failed to compute_task_migrate_server: Timed out waiting for a reply to message ID 237323d0824a43588f6394e5a505248b: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 237323d0824a43588f6394e5a505248b 2021-11-11 15:16:49.122 1887742 WARNING nova.scheduler.utils [req-d648901b-d000-47ec-9497-bcc51586c381 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] [instance: 5935c07d-0c7f-48cc-a4b9-674504fc6005] Setting instance to STOPPED state.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 237323d0824a43588f6394e5a505248b 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server [req-d648901b-d000-47ec-9497-bcc51586c381 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Exception during message handling: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 237323d0824a43588f6394e5a505248b 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 433, in get 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server return self._queues[msg_id].get(block=True, timeout=timeout) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/eventlet/queue.py", line 322, in get 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server return waiter.wait() 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/eventlet/queue.py", line 141, in wait 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server return get_hub().switch() 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 298, in switch 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server return self.greenlet.switch() 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server queue.Empty 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server return func(*args, **kwargs) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 99, in wrapper 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server return fn(self, context, *args, **kwargs) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1434, in decorated_function 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 303, in migrate_server 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server host_list) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 397, in _cold_migrate 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server else 'cold migrate', instance=instance) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server raise value 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 354, in _cold_migrate 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server task.execute() 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/conductor/tasks/base.py", line 26, in wrap 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server self.rollback(ex) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server raise value 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/conductor/tasks/base.py", line 23, in wrap 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server return original(self) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/conductor/tasks/base.py", line 40, in execute 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server return self._execute() 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/conductor/tasks/migrate.py", line 384, in _execute 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server selection = self._schedule() 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/conductor/tasks/migrate.py", line 434, in _schedule 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server return_objects=True, return_alternates=True) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server instance_uuids, return_objects, return_alternates) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server return cctxt.call(ctxt, 'select_destinations', **msg_args) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 179, in call 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server transport_options=self.transport_options) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server transport_options=transport_options) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 682, in send 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server transport_options=transport_options) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 670, in _send 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server call_monitor_timeout) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in wait 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server message = self.waiters.get(msg_id, timeout=timeout) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 437, in get 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server 'to message ID %s' % msg_id) 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 237323d0824a43588f6394e5a505248b
No messages are generated in nova-schedule, or any of the nova-compute logs on the hosts.
Thank you,
Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com