Hello!
Happy new year to everyone! I'm experiencing some trouble in my attempts to get live block migrations working on machines who's boot drives live on ephemeral (local) host storage. Offline (cold) migrations appear to work fine, however, I need live migrations to function with no downtime to the VMs.
We're running on OpenStack Ussuri, with Ubuntu 18.04.5 LTS as our hypervisor OS, running libvirt 6.0.0. Below is the following command I'm utilizing to perform the live block migration:
openstack server migrate <id> --debug --verbose --live-migration --block-migration
The verbose logging of the above command shows no obvious issues, however digging deeper on the hypervisor side, we see some libvirtd errors, and the most telling is this segment here:
Jan 08 16:53:02 h5 libvirtd[4118]: Unsafe migration: Migration without shared storage is unsafe Jan 08 16:53:02 h5 nova-compute[130270]: 2021-01-08 16:53:02.860 130270 ERROR nova.virt.libvirt.driver [-] [instance: 72bc9feb-b8f0-45f2-bad4-9cb61c3abff3] Live Migration failure: Unsafe migration: Migration without shared storage is unsafe: libvirt.libvirtError: Unsafe migration: Migration without shared storage is unsafe
Full log output is available here: http://paste.openstack.org/show/801525/
Perhaps I'm severely mistaken, but native libvirt should have no problem whatsoever performing live migrations without shared storage on it's own. I've dug into the libvirt side for hours to try and troubleshoot what the system is seeing as "unsafe" (I confirmed disk caching is disabled, as well as no strange custom settings are present), but was unsuccessful.
The full output of my VM's QEMU XML file can be viewed here: http://paste.openstack.org/show/801526/
As I couldn't figure out why libvirtd was not seeing the migration as "safe", I attempted to override the libvirt message by attempting to utilize the live_migration_flag and block_migration_flag options in nova.conf to invoke the --unsafe parameter when running the libvirt commands (with the flags VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_UNSAFE,VIR_MIGRATE_NON_SHARED_INC), however I later learned from a previous developers thread on the mailing list from 2016 that both of these options have been depreciated in the later versions of Nova and replaced by the "live_migration_tunnelled" setting in nova.conf.
I've ensured my nova.conf files have the live_migration_tunnelled parameter set to "True". However, I have not found any method for passing along the "--unsafe" flag (or any other libvirt flag, in this case) to libvirt from nova.conf from this point on after the depreciation of the live_migration_flag parameters.
Any help would be greatly appreciated, from reading around online this seems to be a fairly common setup, I'm baffled at how I've run into this issue across several separate OpenStack clusters at this point.
Thank you for reading.