Hello,
looking at destination kvm host I got the following in instance log under /var/log/libvirt/qemu:
2021-03-15 11:48:31.996+0000: starting up libvirt version: 4.5.0, package: 36.el7_9.3 (CentOS BuildSystem <http://bugs.centos.org>, 2020-11-16-16:25:20, x86-01.bsys.centos.org), qemu version: 2.12.0qemu-kvm-ev-2.12.0-44.1.el7_8.1, kernel: 3.10.0-1160.15.2.el7.x86_64, hostname: podto2-kvmae
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
QEMU_AUDIO_DRV=none \
/usr/libexec/qemu-kvm \
-name guest=instance-00002a52,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-73-instance-00002a52/master-key.aes \
-machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off \
-cpu Broadwell-IBRS,vme=on,f16c=on,rdrand=on,hypervisor=on,arat=on,xsaveopt=on,abm=on \
-m 4096 \
-realtime mlock=off \
-smp 2,sockets=2,cores=1,threads=1 \
-uuid c6ea7ed2-e7ce-4df6-a767-6bb95ae8fdc6 \
-smbios 'type=1,manufacturer=RDO,product=OpenStack Compute,version=17.0.11-1.el7,serial=3dec30fe-a31f-4ea6-971f-6f993589ef04,uuid=c6ea7ed2-e7ce-4df6-a767-6bb95ae8fdc6,family=Virtual Machine' \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=139,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 \
-drive file=/var/lib/nova/instances/c6ea7ed2-e7ce-4df6-a767-6bb95ae8fdc6/disk.config,format=raw,if=none,id=drive-ide0-0-0,readonly=on,cache=none \
-device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,write-cache=on \
-drive file=/var/lib/nova/mnt/7eb4b0178ee3ec9ad7cbbc20c62b1912/volume-d5c812c5-2c27-4e82-a38d-83fc79ab848e,format=raw,if=none,id=drive-virtio-disk0,serial=d5c812c5-2c27-4e82-a38d-83fc79ab848e,cache=none,aio=native \
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=on \
-netdev tap,fd=141,id=hostnet0,vhost=on,vhostfd=142 \
-device virtio-net-pci,host_mtu=9000,netdev=hostnet0,id=net0,mac=fa:16:3e:3f:e5:32,bus=pci.0,addr=0x3 \
-add-fd set=3,fd=144 \
-chardev pty,id=charserial0,logfile=/dev/fdset/3,logappend=on \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=143,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-vnc 0.0.0.0:55 \
-k en-us \
-device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
-incoming defer \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2021-03-15 11:48:31.996+0000: Domain id=73 is tainted: high-privileges
2021-03-15T11:48:32.163025Z qemu-kvm: -chardev pty,id=charserial0,logfile=/dev/fdset/3,logappend=on: char device redirected to /dev/pts/57 (label charserial0)
2021-03-15T11:48:32.167206Z qemu-kvm: -drive file=/var/lib/nova/mnt/7eb4b0178ee3ec9ad7cbbc20c62b1912/volume-d5c812c5-2c27-4e82-a38d-83fc79ab848e,format=raw,if=none,id=drive-virtio-disk0,serial=d5c812c5-2c27-4e82-a38d-83fc79ab848e,cache=none,aio=native: 'serial' is deprecated, please use the corresponding option of '-device' instead
2021-03-15T11:48:37.779611Z qemu-kvm: Failed to load virtio_pci/modern_queue_state:desc
2021-03-15T11:48:37.780020Z qemu-kvm: Failed to load virtio_pci/modern_state:vqs
2021-03-15T11:48:37.780042Z qemu-kvm: Failed to load virtio/extra_state:extra_state
2021-03-15T11:48:37.780062Z qemu-kvm: Failed to load virtio-balloon:virtio
2021-03-15T11:48:37.780082Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:06.0/virtio-balloon'
2021-03-15T11:48:37.781465Z qemu-kvm: load of migration failed: Input/output error
2021-03-15 11:48:38.231+0000: shutting down, reason=crashed
"instance-00002a52.log" 102L, 7122C

But hard resetting the vm it starts
Ignazio


Il giorno lun 15 mar 2021 alle ore 17:36 Ignazio Cassano <ignaziocassano@gmail.com> ha scritto:
Hello all,
we are facing a problem while migrating some virtual machines on centos 7 queens.
If we create a new virtual machine it migrates.
Old virtual machines got the following:
37eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Increasing downtime to 50 ms after 0 sec elapsed time
2021-03-15 17:26:44.428 82382 INFO nova.virt.libvirt.driver [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0)
2021-03-15 17:26:57.922 82382 INFO nova.compute.manager [req-83fd3d62-292e-4990-8f7a-47f404b557cc - - - - -] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] VM Paused (Lifecycle Event)
2021-03-15 17:26:58.617 82382 INFO nova.virt.libvirt.driver [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Migration operation has completed
2021-03-15 17:26:58.618 82382 INFO nova.compute.manager [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] _post_live_migration() is started..
2021-03-15 17:26:58.658 82382 ERROR nova.virt.libvirt.driver [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Live Migration failure: operation failed: domain is not running: libvirtError: operation failed: domain is not running
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Post live migration at destination podto2-kvmae failed: InstanceNotFound_Remote: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found.
InstanceNotFound: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found.
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Traceback (most recent call last):
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6505, in _post_live_migration
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     instance, block_migration, dest)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 783, in post_live_migration_at_destination
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     instance=instance, block_migration=block_migration)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     retry=self.retry)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     timeout=timeout, retry=retry)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     retry=retry)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 550, in _send
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     raise result
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] InstanceNotFound_Remote: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found.
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Traceback (most recent call last):
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     res = self.dispatcher.dispatch(message)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     return self._do_dispatch(endpoint, method, ctxt, args)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     result = func(ctxt, **new_args)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     function_name, call_dict, binary)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     self.force_reraise()
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     six.reraise(self.type_, self.value, self.tb)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     return f(self, context, *args, **kw)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1000, in decorated_function
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     return function(self, context, *args, **kwargs)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 203, in decorated_function
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     return function(self, context, *args, **kwargs)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6658, in post_live_migration_at_destination
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     'destination host.', instance=instance)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     self.force_reraise()
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     six.reraise(self.type_, self.value, self.tb)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6653, in post_live_migration_at_destination
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     block_device_info)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7844, in post_live_migration_at_destination
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     self._host.get_guest(instance)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 526, in get_guest
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     return libvirt_guest.Guest(self._get_domain(instance))
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 546, in _get_domain
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]     raise exception.InstanceNotFound(instance_id=instance.uuid)
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] InstanceNotFound: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found.
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572]
2021-03-15 17:27:05.179 82382 WARNING nova.compute.resource_tracker [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Instance not resizing, skipping migration.: InstanceNotFound_Remote: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found.
2021-03-15 17:27:13.621 82382 INFO nova.compute.manager [-] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] VM Stopped (Lifecycle Event)
2021-03-15 17:27:13.711 82382 INFO nova.compute.manager [req-0dcd030c-d520-4223-91a6-f2fbc9f2444a - - - - -] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] During the sync_power process the instance has moved from host podto2-kvmae to host podto2-kvm01


The istances was on podto1-kvm01 and we try to migrate it on podto2-kvmae.
It stops to migrate and goes in error state and stop to responding to ping requests.
From logs is seems it returned on podto2-kvm01 but it is in error state on podto2-kvmae.
After hard rebooting it starts on podto2-kvmae and then we can migrate it where we want without errors.
Please, any help ?
Thanks
Ignazio