Open Stack

It sounds like you've got an issue with your RabbitMQ infrastructure
that's causing messages not to show up.  I suggest focusing your
troubleshooting there.  I've seen these issues resolved by simply
rebuilding the RabbitMQ cluster.

On Thu, Nov 11, 2021 at 6:07 PM <DHilsbos at performair.com> wrote:
>
> All;
>
> I'm running into a strange issue when I try to migrate a server from one host to another.
>
> The only messages I'm seeing are in nova-conductor.log:
> 2021-11-11 15:16:49.118 1887742 WARNING nova.scheduler.utils [req-d648901b-d000-47ec-9497-bcc51586c381 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Failed to compute_task_migrate_server: Timed out waiting for a reply to message ID 237323d0824a43588f6394e5a505248b: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 237323d0824a43588f6394e5a505248b
> 2021-11-11 15:16:49.122 1887742 WARNING nova.scheduler.utils [req-d648901b-d000-47ec-9497-bcc51586c381 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] [instance: 5935c07d-0c7f-48cc-a4b9-674504fc6005] Setting instance to STOPPED state.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 237323d0824a43588f6394e5a505248b
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server [req-d648901b-d000-47ec-9497-bcc51586c381 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Exception during message handling: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 237323d0824a43588f6394e5a505248b
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 433, in get
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return self._queues[msg_id].get(block=True, timeout=timeout)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/queue.py", line 322, in get
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return waiter.wait()
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/queue.py", line 141, in wait
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return get_hub().switch()
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 298, in switch
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return self.greenlet.switch()
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server queue.Empty
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 99, in wrapper
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return fn(self, context, *args, **kwargs)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1434, in decorated_function
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 303, in migrate_server
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     host_list)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 397, in _cold_migrate
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     else 'cold migrate', instance=instance)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     self.force_reraise()
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     raise value
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 354, in _cold_migrate
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     task.execute()
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/conductor/tasks/base.py", line 26, in wrap
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     self.rollback(ex)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     self.force_reraise()
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     raise value
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/conductor/tasks/base.py", line 23, in wrap
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return original(self)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/conductor/tasks/base.py", line 40, in execute
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return self._execute()
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/conductor/tasks/migrate.py", line 384, in _execute
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     selection = self._schedule()
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/conductor/tasks/migrate.py", line 434, in _schedule
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return_objects=True, return_alternates=True)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     instance_uuids, return_objects, return_alternates)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     return cctxt.call(ctxt, 'select_destinations', **msg_args)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 179, in call
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     transport_options=self.transport_options)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     transport_options=transport_options)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 682, in send
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     transport_options=transport_options)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 670, in _send
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     call_monitor_timeout)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in wait
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     message = self.waiters.get(msg_id, timeout=timeout)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 437, in get
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server     'to message ID %s' % msg_id)
> 2021-11-11 15:16:49.291 1887742 ERROR oslo_messaging.rpc.server oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 237323d0824a43588f6394e5a505248b
>
> No messages are generated in nova-schedule, or any of the nova-compute logs on the hosts.
>
> Thank you,
>
> Dominic L. Hilsbos, MBA
> Vice President - Information Technology
> Perform Air International Inc.
> DHilsbos at PerformAir.com
> www.PerformAir.com
>
>
>


-- 
Mohammed Naser
VEXXHOST, Inc.