<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p><font size="-1"><font face="Helvetica, Arial, sans-serif">Hello!</font></font></p>
<p><font size="-1"><font face="Helvetica, Arial, sans-serif">Happy
new year to everyone! I'm experiencing some trouble in my
attempts to get live block migrations working on machines
who's boot drives live on ephemeral (local) host storage.
Offline (cold) migrations appear to work fine, however, I need
live migrations to function with no downtime to the VMs.</font></font></p>
<p><font size="-1"><font face="Helvetica, Arial, sans-serif">We're
running on OpenStack Ussuri, with Ubuntu 18.04.5 LTS as our
hypervisor OS, running libvirt 6.0.0. Below is the following
command I'm utilizing to perform the live block migration:</font></font></p>
<p><font size="-1"><font face="Helvetica, Arial, sans-serif">openstack
server migrate <id> --debug --verbose --live-migration
--block-migration<br>
</font></font></p>
<p><font size="-1"><font face="Helvetica, Arial, sans-serif">The
verbose logging of the above command shows no obvious issues,
however digging deeper on the hypervisor side, we see some
libvirtd errors, and the most telling is this segment here:</font></font></p>
<p><font size="-1"><font face="Helvetica, Arial, sans-serif">Jan 08
16:53:02 h5 libvirtd[4118]: Unsafe migration: Migration
without shared storage is unsafe<br>
Jan 08 16:53:02 h5 nova-compute[130270]: 2021-01-08
16:53:02.860 130270 ERROR nova.virt.libvirt.driver [-]
[instance: 72bc9feb-b8f0-45f2-bad4-9cb61c3abff3] Live
Migration failure: Unsafe migration: Migration without shared
storage is unsafe: libvirt.libvirtError: Unsafe migration:
Migration without shared storage is unsafe</font></font></p>
<p><font size="-1"><font face="Helvetica, Arial, sans-serif">Full
log output is available here: <a moz-do-not-send="true"
href="http://paste.openstack.org/show/801525/">http://paste.openstack.org/show/801525/</a><br>
</font></font></p>
<p><font size="-1"><font face="Helvetica, Arial, sans-serif">Perhaps
I'm severely mistaken, but native libvirt should have no
problem whatsoever performing live migrations without shared
storage on it's own. I've dug into the libvirt side for hours
to try and troubleshoot what the system is seeing as "unsafe"
(I confirmed disk caching is disabled, as well as no strange
custom settings are present), but was unsuccessful. <br>
</font></font></p>
<p><font size="-1"><font face="Helvetica, Arial, sans-serif">The
full output of my VM's QEMU XML file can be viewed here: <a
moz-do-not-send="true"
href="http://paste.openstack.org/show/801526/">http://paste.openstack.org/show/801526/</a><br>
</font></font></p>
<p><font size="-1"><font face="Helvetica, Arial, sans-serif">As I
couldn't figure out why libvirtd was not seeing the migration
as "safe", I attempted to override the libvirt message by
attempting to utilize the live_migration_flag and
block_migration_flag options in nova.conf to invoke the
--unsafe parameter when running the libvirt commands (with the
flags
VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_UNSAFE,VIR_MIGRATE_NON_SHARED_INC),
however I later learned from a previous developers thread on
the mailing list from 2016 that both of these options have
been depreciated in the later versions of Nova and replaced by
the "</font></font><font size="-1"><font face="Helvetica,
Arial, sans-serif"><span id="output">live_migration_tunnelled"
setting in nova.conf.</span></font></font></p>
<p><font size="-1"><font face="Helvetica, Arial, sans-serif"><span
id="output">I've ensured my nova.conf files have the </span></font></font><font
size="-1"><font face="Helvetica, Arial, sans-serif"><span
id="output"><span id="output">live_migration_tunnelled
parameter set to "True". However, I have not found any
method for passing along the "--unsafe" flag (or any other
libvirt flag, in this case) to libvirt from nova.conf from
this point on after the depreciation of the </span></span></font></font><font
size="-1"><font face="Helvetica, Arial, sans-serif"><span
id="output"><span id="output"><font size="-1"><font
face="Helvetica, Arial, sans-serif">live_migration_flag
parameters. <br>
</font></font></span></span></font></font></p>
<p><font size="-1"><font face="Helvetica, Arial, sans-serif"><span
id="output"><span id="output"><font size="-1"><font
face="Helvetica, Arial, sans-serif">Any help would be
greatly appreciated, from reading around online this
seems to be a fairly common setup, I'm baffled at how
I've run into this issue across several separate
OpenStack clusters at this point.</font></font></span></span></font></font></p>
<p><font size="-1"><font face="Helvetica, Arial, sans-serif"><span
id="output"><span id="output"><font size="-1"><font
face="Helvetica, Arial, sans-serif">Thank you for
reading.<br>
</font></font></span></span></font></font></p>
<div class="moz-signature">-- <br>
<p style="color:grey; font-size:small;">Justin Goetz<br>
Systems Engineer, TeraSwitch Inc.<br>
<a class="moz-txt-link-abbreviated" href="mailto:jgoetz@teraswitch.com">jgoetz@teraswitch.com</a> <br>
412-945-7045 (NOC) | 412-459-7945 (Direct)</p>
<img
src="https://support.teraswitch.com/teraswitchbanneremail.png"></div>
</body>
</html>