Open Stack

All;

I'm running into a strange issue when I try to migrate a server from one host to another.

The only messages I'm seeing are in nova-conductor.log:
2021-11-11 15:16:49.118 1887742 WARNING nova.scheduler.utils [req-d648901b-d000-47ec-9497-bcc51586c381 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Failed to compute_task_migrate_server: Timed out waiting for a reply to message ID 237323d0824a43588f6394e5a505248b: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 237323d0824a43588f6394e5a505248b
2021-11-11 15:16:49.122 1887742 WARNING nova.scheduler.utils [req-d648901b-d000-47ec-9497-bcc51586c381 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] [instance: 5935c07d-0c7f-48cc-a4b9-674504fc6005] Setting instance to STOPPED state.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 237323d0824a43588f6394e5a505248b
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server [req-d648901b-d000-47ec-9497-bcc51586c381 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Exception during message handling: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 237323d0824a43588f6394e5a505248b
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 433, in get
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return self._queues[msg_id].get(block=True, timeout=timeout)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/queue.py", line 322, in get
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return waiter.wait()
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/queue.py", line 141, in wait
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return get_hub().switch()
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 298, in switch
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return self.greenlet.switch()
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server queue.Empty
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 99, in wrapper
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return fn(self, context, *args, **kwargs)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1434, in decorated_function
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 303, in migrate_server
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     host_list)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 397, in _cold_migrate
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     else 'cold migrate', instance=instance)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     raise value
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 354, in _cold_migrate
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     task.execute()
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/conductor/tasks/base.py", line 26, in wrap
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     self.rollback(ex)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     raise value
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/conductor/tasks/base.py", line 23, in wrap
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return original(self)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/conductor/tasks/base.py", line 40, in execute
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return self._execute()
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/conductor/tasks/migrate.py", line 384, in _execute
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     selection = self._schedule()
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/conductor/tasks/migrate.py", line 434, in _schedule
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return_objects=True, return_alternates=True)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     instance_uuids, return_objects, return_alternates)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return cctxt.call(ctxt, 'select_destinations', **msg_args)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     transport_options=self.transport_options)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     transport_options=transport_options)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 682, in send
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     transport_options=transport_options)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 670, in _send
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     call_monitor_timeout)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in wait
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     message = self.waiters.get(msg_id, timeout=timeout)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 437, in get
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     'to message ID %s' % msg_id)
2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 237323d0824a43588f6394e5a505248b

No messages are generated in nova-schedule, or any of the nova-compute logs on the hosts.

Thank you,

Dominic L. Hilsbos, MBA
Vice President - Information Technology
Perform Air International Inc.
DHilsbos at PerformAir.com
www.PerformAir.com