<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from rtf -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<font face="Calibri" size="2"><span style="font-size:11pt;">
<div><font color="#1F497D">John;</font></div>
<div><font color="#1F497D"> </font></div>
<div><font color="#1F497D">I recently had to work through a similar issue, though I am working with Victoria, so take this with a grain of salt.</font></div>
<div><font color="#1F497D"> </font></div>
<div><font color="#1F497D">I finally found the correct path by looking in the hypervisor’s logs on the machines sending and receiving the live migration. For us that is KVM.</font></div>
<div><font color="#1F497D"> </font></div>
<div><font color="#1F497D">Thank you,</font></div>
<div><font color="#1F497D"> </font></div>
<div><font size="2" color="#1F497D"><span style="font-size:10.5pt;">Dominic L. Hilsbos, MBA </span></font></div>
<div><font size="2" color="#1F497D"><span style="font-size:10.5pt;">Director – Information Technology </span></font></div>
<div><font size="2" color="#1F497D"><span style="font-size:10.5pt;">Perform Air International Inc.</span></font></div>
<div><font size="2" color="#1F497D"><span style="font-size:10.5pt;">DHilsbos@PerformAir.com </span></font></div>
<div><font size="2" color="#1F497D"><span style="font-size:10.5pt;"><a href="http://www.PerformAir.com">www.PerformAir.com</a></span></font></div>
<div><font color="#1F497D"> </font></div>
<div><font face="Tahoma" size="2"><span style="font-size:10pt;"><b>From:</b> Linebarger, John [<a href="mailto:jmlineb@sandia.gov">mailto:jmlineb@sandia.gov</a>]
<br>
<b>Sent:</b> Tuesday, March 30, 2021 6:24 AM<br>
<b>To:</b> openstack-discuss@lists.openstack.org<br>
<b>Cc:</b> Hostetler, Sarah N; Shurtz, Peter; Urbaniak, Kendrick<br>
<b>Subject:</b> How to debug silent live migration errors</span></font></div>
<div> </div>
<div><font size="4"><span style="font-size:14pt;">How would I debug silent (or mostly silent) live migration errors? We’re using the Stein release of Canonical’s Charmed OpenStack. I have configured it for live migration per the instructions at this link:</span></font></div>
<div> </div>
<div><a href="https://docs.openstack.org/nova/pike/admin/configuring-migrations.html"><font size="4" color="#0563C1"><span style="font-size:14pt;"><u>https://docs.openstack.org/nova/pike/admin/configuring-migrations.html#section-configuring-compute-migrations</u></span></font></a></div>
<div> </div>
<div><font size="4"><span style="font-size:14pt;">Specifically:</span></font></div>
<div><font size="4"><span style="font-size:14pt;"> </span></font></div>
<div><font size="4"><span style="font-size:14pt;">1. I did not specify vncserver_listen=0.0.0.0 in nova.conf because we are not running VNC on our instances</span></font></div>
<div><font size="4"><span style="font-size:14pt;">2. instances_path is /var/lib/nova/instances on all compute nodes</span></font></div>
<div><font size="4"><span style="font-size:14pt;">3. I believe that MAAS is “the sole provider of DHCP and DNS for the network hosting the MAAS cluster”, per <a href="https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/install-maas.html"><font color="#0563C1"><u>https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/install-maas.html</u></font></a></span></font></div>
<div><font size="4"><span style="font-size:14pt;">4. Identical authorized_keys files are present on all compute nodes with keys from all compute nodes by default</span></font></div>
<div><font size="4"><span style="font-size:14pt;">5. I manually configured the firewalls on all compute nodes to allow libvirt to communicate between compute hosts with: </span></font></div>
<div><font face="Courier New" size="2"><span style="font-size:10pt;">sudo ufw allow 49152:49261/tcp</span></font></div>
<div><font size="4"><span style="font-size:14pt;">6. The following settings are specified in nova.conf on each compute node:</span></font></div>
<div><font face="Courier New" size="2"><span style="font-size:10pt;">live_migration_downtime = 500</span></font></div>
<div><font face="Courier New" size="2"><span style="font-size:10pt;">live_migration_downtime_steps = 10</span></font></div>
<div><font face="Courier New" size="2"><span style="font-size:10pt;">live_migration_downtime_delay = 75</span></font></div>
<div><font face="Courier New" size="2"><span style="font-size:10pt;">live_migration_permit_post_copy=true</span></font></div>
<div> </div>
<div><font size="4"><span style="font-size:14pt;">Here’s what happens when I try to Live Migrate from the Horizon Dashboard:</span></font></div>
<div><font size="4"><span style="font-size:14pt;"> </span></font></div>
<div><font size="4"><span style="font-size:14pt;">1. As admin, in the Admin <font face="Wingdings">à</font> Instances menu, I select the dropdown arrow to the right of the instance. Live Migrate Instance appears (but in black, unlike Migrate Instance, which
appears in red). I select Live Migrate Instance, and whether or not I Automatically schedule new host or manually select a new host the Task column says “Migrating” and then it stops and reverts to None. The server never changes. The Action Log shows the live
migration request but the Message column is blank.</span></font></div>
<div><font size="4"><span style="font-size:14pt;"> </span></font></div>
<div><font size="4"><span style="font-size:14pt;">2. I do the very same thing but this time select Disk Over Commit. Same results. Migrating reverts back to None and the server never changes.</span></font></div>
<div><font size="4"><span style="font-size:14pt;"> </span></font></div>
<div><font size="4"><span style="font-size:14pt;">3. I do the very same thing but this time select Block Migration. This time I do get an error: “Failed to live migrate instance to host ‘AUTO_SCHEDULE’”. And this time the Action Log has “Error” in the Message
column.</span></font></div>
<div><font size="4"><span style="font-size:14pt;"> </span></font></div>
<div><font size="4"><span style="font-size:14pt;">Same behavior with the CLI. For example, this CLI command below completes silently, yet the server for the instance never changes.</span></font></div>
<div><font size="4"><span style="font-size:14pt;"> </span></font></div>
<div><font size="4"><span style="font-size:14pt;">john@vm-dev-john:~/bin$ openstack server migrate <instanceID> --live <newServerName></span></font></div>
<div><font size="4"><span style="font-size:14pt;">[Silent failure]</span></font></div>
<div><font size="4"><span style="font-size:14pt;">john@vm-dev-john:~/bin$ openstack server show <instancID></span></font></div>
<div><font size="4"><span style="font-size:14pt;">[Still running on original server]</span></font></div>
<div><font size="4"><span style="font-size:14pt;"> </span></font></div>
<div><font size="4"><span style="font-size:14pt;">Note that I *<b>can</b>* successfully Migrate, both using the Horizon Dashboard and the CLI. What fails is Live Migration. I just have no idea why, and no error is displayed in the Action Log for the instance.</span></font></div>
<div><font size="4"><span style="font-size:14pt;"> </span></font></div>
<div><font size="4"><span style="font-size:14pt;">For reference, the instance is an m1.small with 2GB of RAM, 1 VCPU, and a 20GB Cinder disk volume attached on /dev/vda.</span></font></div>
<div><font size="4"><span style="font-size:14pt;"> </span></font></div>
<div><font size="4"><span style="font-size:14pt;">Any and all debugging ideas would be most welcome. Without logs I am simply guessing in the dark at this point.</span></font></div>
<div><font size="4"><span style="font-size:14pt;"> </span></font></div>
<div><font size="3"><span style="font-size:12pt;">Thanks! Enjoy!</span></font></div>
<div><font size="3"><span style="font-size:12pt;"> </span></font></div>
<div><font size="3"><span style="font-size:12pt;"><b>John M. Linebarger, PhD, MBA</b></span></font></div>
<div><font size="3"><span style="font-size:12pt;">Principal Member of Technical Staff</span></font></div>
<div><font size="3"><span style="font-size:12pt;">Sandia National Laboratories</span></font></div>
<div><font size="3"><span style="font-size:12pt;">(Office) 505-845-8282</span></font></div>
<div><font size="3"><span style="font-size:12pt;">(Cell) 505-681-4879</span></font></div>
<div><a href="http://www.sandia.gov/"><img src="cid:59918635B7DE9E46A5B89B1609A83C36@performair.com"> </a><a href="https://www.youracclaim.com/badges/24fe4e43-2f72-4ecc-a11a-783d21dada0f"><img src="https://www.certmetrics.com/api/ob/image/amazon/c/4"></a><a href="https://www.youracclaim.com/badges/707a5f6f-d919-4daa-bbc4-81b5a779a6cd"><img src="https://www.certmetrics.com/api/ob/image/amazon/c/1"></a><a href="https://www.youracclaim.com/badges/b52ac093-5415-4493-85c3-b77e055211f5"><img src="https://www.certmetrics.com/api/ob/image/amazon/c/2"></a><a href="https://www.youracclaim.com/badges/d6e5a453-0e61-4dbe-9716-6af51f364710/public_url"><img src="cid:03E4CB208B3FE748A93630D26E190D52@performair.com">
</a><a href="https://www.youracclaim.com/badges/035d4967-f77d-476e-915b-1061bb789ec3/public_url"><img src="cid:21F038CE85CDCB448A53D0799CEB03DE@performair.com"> </a></div>
<div> </div>
<div> </div>
</span></font>
</body>
</html>