<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.pre
{mso-style-name:pre;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:14.0pt">How would I debug silent (or mostly silent) live migration errors? We’re using the Stein release of Canonical’s Charmed OpenStack. I have configured it for live migration per the instructions at this link:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><a href="https://docs.openstack.org/nova/pike/admin/configuring-migrations.html#section-configuring-compute-migrations">https://docs.openstack.org/nova/pike/admin/configuring-migrations.html#section-configuring-compute-migrations</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">Specifically:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">1. I did not specify vncserver_listen=0.0.0.0 in nova.conf because we are not running VNC on our instances<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">2. instances_path is /var/lib/nova/instances on all compute nodes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">3. I believe that MAAS is “the sole provider of DHCP and DNS for the network hosting the MAAS cluster”, per
<a href="https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/install-maas.html">
https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/install-maas.html</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">4. Identical authorized_keys files are present on all compute nodes with keys from all compute nodes by default<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">5. I manually configured the firewalls on all compute nodes to allow libvirt to communicate between compute hosts with:
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">sudo ufw allow 49152:49261/tcp</span><span style="font-size:14.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">6. The following settings are specified in nova.conf on each compute node:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">live_migration_downtime = 500<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">live_migration_downtime_steps = 10<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">live_migration_downtime_delay = 75<o:p></o:p></span></p>
<p class="MsoNormal"><span class="pre"><span style="font-size:10.0pt;font-family:"Courier New"">live_migration_permit_post_copy=true</span></span><span style="font-size:14.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">Here’s what happens when I try to Live Migrate from the Horizon Dashboard:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">1. As admin, in the Admin </span>
<span style="font-size:14.0pt;font-family:Wingdings">à</span><span style="font-size:14.0pt"> Instances menu, I select the dropdown arrow to the right of the instance. Live Migrate Instance appears (but in black, unlike Migrate Instance, which appears in red).
I select Live Migrate Instance, and whether or not I Automatically schedule new host or manually select a new host the Task column says “Migrating” and then it stops and reverts to None. The server never changes. The Action Log shows the live migration request
but the Message column is blank.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">2. I do the very same thing but this time select Disk Over Commit. Same results. Migrating reverts back to None and the server never changes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">3. I do the very same thing but this time select Block Migration. This time I do get an error: “Failed to live migrate instance to host ‘AUTO_SCHEDULE’”. And this time the Action Log has “Error” in the Message
column.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">Same behavior with the CLI. For example, this CLI command below completes silently, yet the server for the instance never changes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">john@vm-dev-john:~/bin$ openstack server migrate <instanceID> --live <newServerName><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">[Silent failure]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">john@vm-dev-john:~/bin$ openstack server show <instancID><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">[Still running on original server]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">Note that I *<b>can</b>* successfully Migrate, both using the Horizon Dashboard and the CLI. What fails is Live Migration. I just have no idea why, and no error is displayed in the Action Log for the instance.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">For reference, the instance is an m1.small with 2GB of RAM, 1 VCPU, and a 20GB Cinder disk volume attached on /dev/vda.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">Any and all debugging ideas would be most welcome. Without logs I am simply guessing in the dark at this point.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">Thanks! Enjoy!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:12.0pt">John M. Linebarger, PhD, MBA<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:12.0pt">Principal Member of Technical Staff<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">Sandia National Laboratories<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">(Office) 505-845-8282<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">(Cell) 505-681-4879<o:p></o:p></span></p>
<p class="MsoNormal"><a href="http://www.sandia.gov/"><span style="color:windowtext;text-decoration:none"><img border="0" width="77" height="77" style="width:.802in;height:.802in" id="Picture_x0020_10" src="cid:image002.jpg@01D72535.958BCAE0"></span></a><a href="https://www.youracclaim.com/badges/24fe4e43-2f72-4ecc-a11a-783d21dada0f"><span style="color:blue;border:none windowtext 1.0pt;padding:0in;text-decoration:none"><img border="0" width="80" height="80" style="width:.8333in;height:.8333in" id="_x0000_i1029" src="https://www.certmetrics.com/api/ob/image/amazon/c/4" alt="AWS Certified Solutions Architect - Professional"></span></a><a href="https://www.youracclaim.com/badges/707a5f6f-d919-4daa-bbc4-81b5a779a6cd"><span style="color:blue;border:none windowtext 1.0pt;padding:0in;text-decoration:none"><img border="0" width="80" height="80" style="width:.8333in;height:.8333in" id="_x0000_i1028" src="https://www.certmetrics.com/api/ob/image/amazon/c/1" alt="AWS Certified Solutions Architect - Associate"></span></a><a href="https://www.youracclaim.com/badges/b52ac093-5415-4493-85c3-b77e055211f5"><span style="color:blue;border:none windowtext 1.0pt;padding:0in;text-decoration:none"><img border="0" width="80" height="80" style="width:.8333in;height:.8333in" id="_x0000_i1027" src="https://www.certmetrics.com/api/ob/image/amazon/c/2" alt="AWS Certified Developer - Associate"></span></a><a href="https://www.youracclaim.com/badges/d6e5a453-0e61-4dbe-9716-6af51f364710/public_url"><span style="color:windowtext;text-decoration:none"><img border="0" width="79" height="79" style="width:.8229in;height:.8229in" id="Picture_x0020_5" src="cid:image003.png@01D72531.072F13F0"></span></a><a href="https://www.youracclaim.com/badges/035d4967-f77d-476e-915b-1061bb789ec3/public_url"><span style="color:windowtext;text-decoration:none"><img border="0" width="92" height="77" style="width:.9583in;height:.802in" id="Picture_x0020_12" src="cid:image005.png@01D72535.958BCAE0"></span></a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>