Open Stack

Hello all,
we are facing a problem while migrating some virtual machines on centos 7
queens.
If we create a new virtual machine it migrates.
Old virtual machines got the following:
37eb1d0c4c - default default] [instance:
25fbb55c-5991-49b2-885f-26adebaeb572] Increasing downtime to 50 ms after 0
sec elapsed time
2021-03-15 17:26:44.428 82382 INFO nova.virt.libvirt.driver
[req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca
85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance:
25fbb55c-5991-49b2-885f-26adebaeb572] Migration running for 0 secs, memory
100% remaining; (bytes processed=0, remaining=0, total=0)
2021-03-15 17:26:57.922 82382 INFO nova.compute.manager
[req-83fd3d62-292e-4990-8f7a-47f404b557cc - - - - -] [instance:
25fbb55c-5991-49b2-885f-26adebaeb572] VM Paused (Lifecycle Event)
2021-03-15 17:26:58.617 82382 INFO nova.virt.libvirt.driver
[req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca
85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance:
25fbb55c-5991-49b2-885f-26adebaeb572] Migration operation has completed
2021-03-15 17:26:58.618 82382 INFO nova.compute.manager
[req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca
85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance:
25fbb55c-5991-49b2-885f-26adebaeb572] _post_live_migration() is started..
2021-03-15 17:26:58.658 82382 ERROR nova.virt.libvirt.driver
[req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca
85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance:
25fbb55c-5991-49b2-885f-26adebaeb572] Live Migration failure: operation
failed: domain is not running: libvirtError: operation failed: domain is
not running
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager
[req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca
85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance:
25fbb55c-5991-49b2-885f-26adebaeb572] Post live migration at destination
podto2-kvmae failed: InstanceNotFound_Remote: Instance
25fbb55c-5991-49b2-885f-26adebaeb572 could not be found.
InstanceNotFound: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not
be found.
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572] Traceback (most recent call last):
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6505, in
_post_live_migration
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     instance, block_migration, dest)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 783, in
post_live_migration_at_destination
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     instance=instance,
block_migration=block_migration)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174,
in call
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     retry=self.retry)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131,
in _send
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     timeout=timeout, retry=retry)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 559, in send
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     retry=retry)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 550, in _send
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     raise result
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572] InstanceNotFound_Remote: Instance
25fbb55c-5991-49b2-885f-26adebaeb572 could not be found.
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572] Traceback (most recent call last):
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166,
in _process_incoming
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     res =
self.dispatcher.dispatch(message)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line
220, in dispatch
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     return
self._do_dispatch(endpoint, method, ctxt, args)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line
190, in _do_dispatch
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     result = func(ctxt, **new_args)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in
wrapped
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     function_name, call_dict, binary)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
__exit__
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     self.force_reraise()
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     six.reraise(self.type_,
self.value, self.tb)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in
wrapped
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     return f(self, context, *args,
**kw)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1000, in
decorated_function
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     return function(self, context,
*args, **kwargs)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 203, in
decorated_function
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     return function(self, context,
*args, **kwargs)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6658, in
post_live_migration_at_destination
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     'destination host.',
instance=instance)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
__exit__
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     self.force_reraise()
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     six.reraise(self.type_,
self.value, self.tb)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6653, in
post_live_migration_at_destination
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     block_device_info)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7844,
in post_live_migration_at_destination
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     self._host.get_guest(instance)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 526, in
get_guest
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     return
libvirt_guest.Guest(self._get_domain(instance))
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]   File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 546, in
_get_domain
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]     raise
exception.InstanceNotFound(instance_id=instance.uuid)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572] InstanceNotFound: Instance
25fbb55c-5991-49b2-885f-26adebaeb572 could not be found.
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:05.179 82382 WARNING nova.compute.resource_tracker
[req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca
85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance:
25fbb55c-5991-49b2-885f-26adebaeb572] Instance not resizing, skipping
migration.: InstanceNotFound_Remote: Instance
25fbb55c-5991-49b2-885f-26adebaeb572 could not be found.
2021-03-15 17:27:13.621 82382 INFO nova.compute.manager [-] [instance:
25fbb55c-5991-49b2-885f-26adebaeb572] VM Stopped (Lifecycle Event)
2021-03-15 17:27:13.711 82382 INFO nova.compute.manager
[req-0dcd030c-d520-4223-91a6-f2fbc9f2444a - - - - -] [instance:
25fbb55c-5991-49b2-885f-26adebaeb572] During the sync_power process the
instance has moved from host podto2-kvmae to host podto2-kvm01


The istances was on podto1-kvm01 and we try to migrate it on podto2-kvmae.
It stops to migrate and goes in error state and stop to responding to ping
requests.
>From logs is seems it returned on podto2-kvm01 but it is in error state on
podto2-kvmae.
After hard rebooting it starts on podto2-kvmae and then we can migrate it
where we want without errors.
Please, any help ?
Thanks
Ignazio
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210315/42ecdb09/attachment-0001.html>