Open Stack

Hello!

Happy new year to everyone! I'm experiencing some trouble in my attempts 
to get live block migrations working on machines who's boot drives live 
on ephemeral (local) host storage. Offline (cold) migrations appear to 
work fine, however, I need live migrations to function with no downtime 
to the VMs.

We're running on OpenStack Ussuri, with Ubuntu 18.04.5 LTS as our 
hypervisor OS, running libvirt 6.0.0. Below is the following command I'm 
utilizing to perform the live block migration:

openstack server migrate <id> --debug --verbose --live-migration 
--block-migration

The verbose logging of the above command shows no obvious issues, 
however digging deeper on the hypervisor side, we see some libvirtd 
errors, and the most telling is this segment here:

Jan 08 16:53:02 h5 libvirtd[4118]: Unsafe migration: Migration without 
shared storage is unsafe
Jan 08 16:53:02 h5 nova-compute[130270]: 2021-01-08 16:53:02.860 130270 
ERROR nova.virt.libvirt.driver [-] [instance: 
72bc9feb-b8f0-45f2-bad4-9cb61c3abff3] Live Migration failure: Unsafe 
migration: Migration without shared storage is unsafe: 
libvirt.libvirtError: Unsafe migration: Migration without shared storage 
is unsafe

Full log output is available here: http://paste.openstack.org/show/801525/

Perhaps I'm severely mistaken, but native libvirt should have no problem 
whatsoever performing live migrations without shared storage on it's 
own. I've dug into the libvirt side for hours to try and troubleshoot 
what the system is seeing as "unsafe" (I confirmed disk caching is 
disabled, as well as no strange custom settings are present), but was 
unsuccessful.

The full output of my VM's QEMU XML file can be viewed here: 
http://paste.openstack.org/show/801526/

As I couldn't figure out why libvirtd was not seeing the migration as 
"safe", I attempted to override the libvirt message by attempting to 
utilize the live_migration_flag and block_migration_flag options in 
nova.conf to invoke the --unsafe parameter when running the libvirt 
commands (with the flags 
VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_UNSAFE,VIR_MIGRATE_NON_SHARED_INC), 
however I later learned from a previous developers thread on the mailing 
list from 2016 that both of these options have been depreciated in the 
later versions of Nova and replaced by the "live_migration_tunnelled" 
setting in nova.conf.

I've ensured my nova.conf files have the live_migration_tunnelled 
parameter set to "True". However, I have not found any method for 
passing along the "--unsafe" flag (or any other libvirt flag, in this 
case) to libvirt from nova.conf from this point on after the 
depreciation of the live_migration_flag parameters.

Any help would be greatly appreciated, from reading around online this 
seems to be a fairly common setup, I'm baffled at how I've run into this 
issue across several separate OpenStack clusters at this point.

Thank you for reading.

-- 

Justin Goetz
Systems Engineer, TeraSwitch Inc.
jgoetz at teraswitch.com
412-945-7045 (NOC) | 412-459-7945 (Direct)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210108/7417d53e/attachment-0001.html>