[nova] [Ussuri] Unable to perform live block migrations on ephemeral VMs without shared storage
jgoetz at teraswitch.com
Fri Jan 8 22:19:25 UTC 2021
Happy new year to everyone! I'm experiencing some trouble in my attempts
to get live block migrations working on machines who's boot drives live
on ephemeral (local) host storage. Offline (cold) migrations appear to
work fine, however, I need live migrations to function with no downtime
to the VMs.
We're running on OpenStack Ussuri, with Ubuntu 18.04.5 LTS as our
hypervisor OS, running libvirt 6.0.0. Below is the following command I'm
utilizing to perform the live block migration:
openstack server migrate <id> --debug --verbose --live-migration
The verbose logging of the above command shows no obvious issues,
however digging deeper on the hypervisor side, we see some libvirtd
errors, and the most telling is this segment here:
Jan 08 16:53:02 h5 libvirtd: Unsafe migration: Migration without
shared storage is unsafe
Jan 08 16:53:02 h5 nova-compute: 2021-01-08 16:53:02.860 130270
ERROR nova.virt.libvirt.driver [-] [instance:
72bc9feb-b8f0-45f2-bad4-9cb61c3abff3] Live Migration failure: Unsafe
migration: Migration without shared storage is unsafe:
libvirt.libvirtError: Unsafe migration: Migration without shared storage
Full log output is available here: http://paste.openstack.org/show/801525/
Perhaps I'm severely mistaken, but native libvirt should have no problem
whatsoever performing live migrations without shared storage on it's
own. I've dug into the libvirt side for hours to try and troubleshoot
what the system is seeing as "unsafe" (I confirmed disk caching is
disabled, as well as no strange custom settings are present), but was
The full output of my VM's QEMU XML file can be viewed here:
As I couldn't figure out why libvirtd was not seeing the migration as
"safe", I attempted to override the libvirt message by attempting to
utilize the live_migration_flag and block_migration_flag options in
nova.conf to invoke the --unsafe parameter when running the libvirt
commands (with the flags
however I later learned from a previous developers thread on the mailing
list from 2016 that both of these options have been depreciated in the
later versions of Nova and replaced by the "live_migration_tunnelled"
setting in nova.conf.
I've ensured my nova.conf files have the live_migration_tunnelled
parameter set to "True". However, I have not found any method for
passing along the "--unsafe" flag (or any other libvirt flag, in this
case) to libvirt from nova.conf from this point on after the
depreciation of the live_migration_flag parameters.
Any help would be greatly appreciated, from reading around online this
seems to be a fairly common setup, I'm baffled at how I've run into this
issue across several separate OpenStack clusters at this point.
Thank you for reading.
Systems Engineer, TeraSwitch Inc.
jgoetz at teraswitch.com
412-945-7045 (NOC) | 412-459-7945 (Direct)
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the openstack-discuss