Hello, looking at destination kvm host I got the following in instance log under /var/log/libvirt/qemu: 2021-03-15 11:48:31.996+0000: starting up libvirt version: 4.5.0, package: 36.el7_9.3 (CentOS BuildSystem <http://bugs.centos.org>, 2020-11-16-16:25:20, x86-01.bsys.centos.org), qemu version: 2.12.0qemu-kvm-ev-2.12.0-44.1.el7_8.1, kernel: 3.10.0-1160.15.2.el7.x86_64, hostname: podto2-kvmae LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \ QEMU_AUDIO_DRV=none \ /usr/libexec/qemu-kvm \ -name guest=instance-00002a52,debug-threads=on \ -S \ -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-73-instance-00002a52/master-key.aes \ -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off \ -cpu Broadwell-IBRS,vme=on,f16c=on,rdrand=on,hypervisor=on,arat=on,xsaveopt=on,abm=on \ -m 4096 \ -realtime mlock=off \ -smp 2,sockets=2,cores=1,threads=1 \ -uuid c6ea7ed2-e7ce-4df6-a767-6bb95ae8fdc6 \ -smbios 'type=1,manufacturer=RDO,product=OpenStack Compute,version=17.0.11-1.el7,serial=3dec30fe-a31f-4ea6-971f-6f993589ef04,uuid=c6ea7ed2-e7ce-4df6-a767-6bb95ae8fdc6,family=Virtual Machine' \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=139,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc,driftfix=slew \ -global kvm-pit.lost_tick_policy=delay \ -no-hpet \ -no-shutdown \ -boot strict=on \ -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \ -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 \ -drive file=/var/lib/nova/instances/c6ea7ed2-e7ce-4df6-a767-6bb95ae8fdc6/disk.config,format=raw,if=none,id=drive-ide0-0-0,readonly=on,cache=none \ -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,write-cache=on \ -drive file=/var/lib/nova/mnt/7eb4b0178ee3ec9ad7cbbc20c62b1912/volume-d5c812c5-2c27-4e82-a38d-83fc79ab848e,format=raw,if=none,id=drive-virtio-disk0,serial=d5c812c5-2c27-4e82-a38d-83fc79ab848e,cache=none,aio=native \ -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=on \ -netdev tap,fd=141,id=hostnet0,vhost=on,vhostfd=142 \ -device virtio-net-pci,host_mtu=9000,netdev=hostnet0,id=net0,mac=fa:16:3e:3f:e5:32,bus=pci.0,addr=0x3 \ -add-fd set=3,fd=144 \ -chardev pty,id=charserial0,logfile=/dev/fdset/3,logappend=on \ -device isa-serial,chardev=charserial0,id=serial0 \ -chardev socket,id=charchannel0,fd=143,server,nowait \ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \ -device usb-tablet,id=input0,bus=usb.0,port=1 \ -vnc 0.0.0.0:55 \ -k en-us \ -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \ -incoming defer \ -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on 2021-03-15 11:48:31.996+0000: Domain id=73 is tainted: high-privileges 2021-03-15T11:48:32.163025Z qemu-kvm: -chardev pty,id=charserial0,logfile=/dev/fdset/3,logappend=on: char device redirected to /dev/pts/57 (label charserial0) 2021-03-15T11:48:32.167206Z qemu-kvm: -drive file=/var/lib/nova/mnt/7eb4b0178ee3ec9ad7cbbc20c62b1912/volume-d5c812c5-2c27-4e82-a38d-83fc79ab848e,format=raw,if=none,id=drive-virtio-disk0,serial=d5c812c5-2c27-4e82-a38d-83fc79ab848e,cache=none,aio=native: 'serial' is deprecated, please use the corresponding option of '-device' instead 2021-03-15T11:48:37.779611Z qemu-kvm: Failed to load virtio_pci/modern_queue_state:desc 2021-03-15T11:48:37.780020Z qemu-kvm: Failed to load virtio_pci/modern_state:vqs 2021-03-15T11:48:37.780042Z qemu-kvm: Failed to load virtio/extra_state:extra_state 2021-03-15T11:48:37.780062Z qemu-kvm: Failed to load virtio-balloon:virtio 2021-03-15T11:48:37.780082Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:06.0/virtio-balloon' 2021-03-15T11:48:37.781465Z qemu-kvm: load of migration failed: Input/output error 2021-03-15 11:48:38.231+0000: shutting down, reason=crashed "instance-00002a52.log" 102L, 7122C But hard resetting the vm it starts Ignazio Il giorno lun 15 mar 2021 alle ore 17:36 Ignazio Cassano < ignaziocassano@gmail.com> ha scritto:
Hello all, we are facing a problem while migrating some virtual machines on centos 7 queens. If we create a new virtual machine it migrates. Old virtual machines got the following: 37eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Increasing downtime to 50 ms after 0 sec elapsed time 2021-03-15 17:26:44.428 82382 INFO nova.virt.libvirt.driver [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0) 2021-03-15 17:26:57.922 82382 INFO nova.compute.manager [req-83fd3d62-292e-4990-8f7a-47f404b557cc - - - - -] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] VM Paused (Lifecycle Event) 2021-03-15 17:26:58.617 82382 INFO nova.virt.libvirt.driver [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Migration operation has completed 2021-03-15 17:26:58.618 82382 INFO nova.compute.manager [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] _post_live_migration() is started.. 2021-03-15 17:26:58.658 82382 ERROR nova.virt.libvirt.driver [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Live Migration failure: operation failed: domain is not running: libvirtError: operation failed: domain is not running 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Post live migration at destination podto2-kvmae failed: InstanceNotFound_Remote: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found. InstanceNotFound: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found. 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Traceback (most recent call last): 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6505, in _post_live_migration 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] instance, block_migration, dest) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 783, in post_live_migration_at_destination 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] instance=instance, block_migration=block_migration) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] retry=self.retry) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] timeout=timeout, retry=retry) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] retry=retry) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 550, in _send 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] raise result 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] InstanceNotFound_Remote: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found. 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Traceback (most recent call last): 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] res = self.dispatcher.dispatch(message) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] return self._do_dispatch(endpoint, method, ctxt, args) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] result = func(ctxt, **new_args) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] function_name, call_dict, binary) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] self.force_reraise() 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] six.reraise(self.type_, self.value, self.tb) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] return f(self, context, *args, **kw) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1000, in decorated_function 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] return function(self, context, *args, **kwargs) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 203, in decorated_function 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] return function(self, context, *args, **kwargs) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6658, in post_live_migration_at_destination 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 'destination host.', instance=instance) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] self.force_reraise() 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] six.reraise(self.type_, self.value, self.tb) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6653, in post_live_migration_at_destination 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] block_device_info) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7844, in post_live_migration_at_destination 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] self._host.get_guest(instance) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 526, in get_guest 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] return libvirt_guest.Guest(self._get_domain(instance)) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 546, in _get_domain 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] raise exception.InstanceNotFound(instance_id=instance.uuid) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] InstanceNotFound: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found. 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:05.179 82382 WARNING nova.compute.resource_tracker [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Instance not resizing, skipping migration.: InstanceNotFound_Remote: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found. 2021-03-15 17:27:13.621 82382 INFO nova.compute.manager [-] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] VM Stopped (Lifecycle Event) 2021-03-15 17:27:13.711 82382 INFO nova.compute.manager [req-0dcd030c-d520-4223-91a6-f2fbc9f2444a - - - - -] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] During the sync_power process the instance has moved from host podto2-kvmae to host podto2-kvm01
The istances was on podto1-kvm01 and we try to migrate it on podto2-kvmae. It stops to migrate and goes in error state and stop to responding to ping requests. From logs is seems it returned on podto2-kvm01 but it is in error state on podto2-kvmae. After hard rebooting it starts on podto2-kvmae and then we can migrate it where we want without errors. Please, any help ? Thanks Ignazio