<div dir="ltr"><div>Hello all,</div><div>we are facing a problem while migrating some virtual machines on centos 7 queens.</div><div>If we create a new virtual machine it migrates.</div><div>Old virtual machines got the following:</div><div>37eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Increasing downtime to 50 ms after 0 sec elapsed time<br>2021-03-15 17:26:44.428 82382 INFO nova.virt.libvirt.driver [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0)<br>2021-03-15 17:26:57.922 82382 INFO nova.compute.manager [req-83fd3d62-292e-4990-8f7a-47f404b557cc - - - - -] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] VM Paused (Lifecycle Event)<br>2021-03-15 17:26:58.617 82382 INFO nova.virt.libvirt.driver [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Migration operation has completed<br>2021-03-15 17:26:58.618 82382 INFO nova.compute.manager [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] _post_live_migration() is started..<br>2021-03-15 17:26:58.658 82382 ERROR nova.virt.libvirt.driver [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Live Migration failure: operation failed: domain is not running: libvirtError: operation failed: domain is not running<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Post live migration at destination podto2-kvmae failed: InstanceNotFound_Remote: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found.<br>InstanceNotFound: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found.<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Traceback (most recent call last):<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6505, in _post_live_migration<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     instance, block_migration, dest)<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 783, in post_live_migration_at_destination<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     instance=instance, block_migration=block_migration)<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     retry=self.retry)<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     timeout=timeout, retry=retry)<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     retry=retry)<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 550, in _send<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     raise result<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] InstanceNotFound_Remote: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found.<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Traceback (most recent call last):<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     res = self.dispatcher.dispatch(message)<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     return self._do_dispatch(endpoint, method, ctxt, args)<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     result = func(ctxt, **new_args)<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     function_name, call_dict, binary)<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     self.force_reraise()<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     six.reraise(self.type_, self.value, self.tb)<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     return f(self, context, *args, **kw)<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1000, in decorated_function<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     return function(self, context, *args, **kwargs)<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 203, in decorated_function<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     return function(self, context, *args, **kwargs)<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6658, in post_live_migration_at_destination<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     'destination host.', instance=instance)<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     self.force_reraise()<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     six.reraise(self.type_, self.value, self.tb)<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6653, in post_live_migration_at_destination<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     block_device_info)<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7844, in post_live_migration_at_destination<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     self._host.get_guest(instance)<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 526, in get_guest<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     return libvirt_guest.Guest(self._get_domain(instance))<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 546, in _get_domain<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     raise exception.InstanceNotFound(instance_id=instance.uuid)<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] InstanceNotFound: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found.<br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] <br>2021-03-15 17:27:05.179 82382 WARNING nova.compute.resource_tracker [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Instance not resizing, skipping migration.: InstanceNotFound_Remote: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found.<br>2021-03-15 17:27:13.621 82382 INFO nova.compute.manager [-] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] VM Stopped (Lifecycle Event)<br>2021-03-15 17:27:13.711 82382 INFO nova.compute.manager [req-0dcd030c-d520-4223-91a6-f2fbc9f2444a - - - - -] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] During the sync_power process the instance has moved from host podto2-kvmae to host podto2-kvm01</div><div><br></div><div><br></div><div>The istances was on podto1-kvm01 and we try to migrate it on podto2-kvmae.</div><div>It stops to migrate and goes in error state and stop to responding to ping requests.</div><div>From logs is seems it returned on podto2-kvm01 but it is in error state on podto2-kvmae.</div><div>After hard rebooting it starts on podto2-kvmae and then we can migrate it where we want without errors.</div><div>Please, any help ?</div><div>Thanks</div><div>Ignazio<br></div></div>