<div dir="ltr">Hi guys, last friday i started testing live-migration in my grizzly cloud with shared storage (gfs2) but i run into a problem, a little weird:<div><br></div><div>This is the status before migrating:</div><div>
<br></div><div>-I've p9 instances also called instance-00000022 running on "acelga" compute node.</div><div><br></div><div><div><i><font face="courier new, monospace">root@acelga:~/tools# virsh list</font></i></div>
<div><i><font face="courier new, monospace"> Id Name State</font></i></div><div><i><font face="courier new, monospace">----------------------------------------------------</font></i></div><div>
<i><font face="courier new, monospace"> 6 instance-00000022 running</font></i></div><div><i><font face="courier new, monospace"><br></font></i></div><div><i><font face="courier new, monospace">root@acelga:~/tools# </font></i></div>
<div><i><font face="courier new, monospace"><br></font></i></div><div><i><font face="courier new, monospace"><br></font></i></div><div><i><font face="courier new, monospace">root@cebolla:~/tool# virsh list</font></i></div>
<div><i><font face="courier new, monospace"> Id Nombre Estado</font></i></div><div><i><font face="courier new, monospace">----------------------------------------------------</font></i></div><div>
<i><font face="courier new, monospace"><br></font></i></div><div><i><font face="courier new, monospace">root@cebolla:~/tool# </font></i></div><div><br></div><div>-Here you can see all the info about the instance</div><div>
<br></div><div><font face="courier new, monospace"><i>root@cebolla:~/tool# nova --os-username=noc-admin --os-tenant-name=noc --os-password=XXXXXXX --os-auth-url <a href="http://172.19.136.1:35357/v2.0">http://172.19.136.1:35357/v2.0</a> show de2bcbed-f7b6-40cd-89ca-acf6fe2f2d09</i></font></div>
<div><font face="courier new, monospace"><i>+-------------------------------------+-----------------------------------------------------------+</i></font></div><div><font face="courier new, monospace"><i>| Property | Value |</i></font></div>
<div><font face="courier new, monospace"><i>+-------------------------------------+-----------------------------------------------------------+</i></font></div><div><font face="courier new, monospace"><i>| status | ACTIVE |</i></font></div>
<div><font face="courier new, monospace"><i>| updated | 2013-09-02T15:27:39Z |</i></font></div><div><font face="courier new, monospace"><i>| OS-EXT-STS:task_state | None |</i></font></div>
<div><font face="courier new, monospace"><i>| OS-EXT-SRV-ATTR:host | acelga |</i></font></div><div><font face="courier new, monospace"><i>| key_name | None |</i></font></div>
<div><font face="courier new, monospace"><i>| image | Ubuntu 12.04.2 LTS (1359ca8d-23a2-40e8-940f-d90b3e68bb39) |</i></font></div><div><font face="courier new, monospace"><i>| vlan1 network | 172.16.16.175 |</i></font></div>
<div><font face="courier new, monospace"><i>| hostId | 81be94870821e17e327d92e9c80548ffcdd37d24054a235116669f53 |</i></font></div><div><font face="courier new, monospace"><i>| OS-EXT-STS:vm_state | active |</i></font></div>
<div><font face="courier new, monospace"><i>| OS-EXT-SRV-ATTR:instance_name | instance-00000022 |</i></font></div><div><font face="courier new, monospace"><i>| OS-EXT-SRV-ATTR:hypervisor_hostname | <a href="http://acelga.psi.unc.edu.ar">acelga.psi.unc.edu.ar</a> |</i></font></div>
<div><font face="courier new, monospace"><i>| flavor | m1.tiny (1) |</i></font></div><div><font face="courier new, monospace"><i>| id | de2bcbed-f7b6-40cd-89ca-acf6fe2f2d09 |</i></font></div>
<div><font face="courier new, monospace"><i>| security_groups | [{u'name': u'default'}] |</i></font></div><div><font face="courier new, monospace"><i>| user_id | 20390b639d4449c18926dca5e038ec5e |</i></font></div>
<div><font face="courier new, monospace"><i>| name | p9 |</i></font></div><div><font face="courier new, monospace"><i>| created | 2013-09-02T15:27:06Z |</i></font></div>
<div><font face="courier new, monospace"><i>| tenant_id | d1e3aae242f14c488d2225dcbf1e96d6 |</i></font></div><div><font face="courier new, monospace"><i>| OS-DCF:diskConfig | MANUAL |</i></font></div>
<div><font face="courier new, monospace"><i>| metadata | {} |</i></font></div><div><font face="courier new, monospace"><i>| accessIPv4 | |</i></font></div>
<div><font face="courier new, monospace"><i>| accessIPv6 | |</i></font></div><div><font face="courier new, monospace"><i>| progress | 0 |</i></font></div>
<div><font face="courier new, monospace"><i>| OS-EXT-STS:power_state | 1 |</i></font></div><div><font face="courier new, monospace"><i>| OS-EXT-AZ:availability_zone | nova |</i></font></div>
<div><font face="courier new, monospace"><i>| config_drive | |</i></font></div><div><font face="courier new, monospace"><i>+-------------------------------------+-----------------------------------------------------------+</i></font></div>
<div><font face="courier new, monospace"><i>root@cebolla:~/tool#</i></font></div><div><br></div><div>-So i try to move it to the other node "cebolla"</div><div><br></div><div><font face="courier new, monospace"><i>root@acelga:~/tools# nova --os-username=noc-admin --os-tenant-name=noc --os-password=HjZ5V9yj --os-auth-url <a href="http://172.19.136.1:35357/v2.0">http://172.19.136.1:35357/v2.0</a> live-migration de2bcbed-f7b6-40cd-89ca-acf6fe2f2d09 cebolla<br>
</i></font></div><div><div><font face="courier new, monospace"><i>root@acelga:~/tools# virsh list</i></font></div><div><font face="courier new, monospace"><i> Id Name State</i></font></div><div>
<font face="courier new, monospace"><i>----------------------------------------------------</i></font></div><div><font face="courier new, monospace"><i><br></i></font></div><div><font face="courier new, monospace"><i>root@acelga:~/tools#</i></font></div>
</div><div><br></div><div>No error messages at all on "acelga" compute node so far. If i check the other node i can see the instance've been migrated</div><div><br></div><div><div><i><font face="courier new, monospace">root@cebolla:~/tool# virsh list</font></i></div>
<div><i><font face="courier new, monospace"> Id Nombre Estado</font></i></div><div><i><font face="courier new, monospace">----------------------------------------------------</font></i></div><div>
<i><font face="courier new, monospace"> 11 instance-00000022 ejecutando</font></i></div><div><i><font face="courier new, monospace"><br></font></i></div><div><i><font face="courier new, monospace">root@cebolla:~/tool#</font></i> </div>
</div><div><br></div><div><br></div><div>-BUT... after a few seconds i get this on "acelga"'s nova-compute.log</div><div><br></div><div><br></div><div><div><i><font face="courier new, monospace">2013-09-02 15:35:45.784 4601 DEBUG nova.openstack.common.rpc.common [-] Timed out waiting for RPC response: timed out _error_callback /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py:628</font></i></div>
<div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 ERROR nova.utils [-] in fixed duration looping call</font></i></div><div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils Traceback (most recent call last):</font></i></div>
<div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 594, in _inner</font></i></div><div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils self.f(*self.args, **<a href="http://self.kw">self.kw</a>)</font></i></div>
<div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3129, in wait_for_live_migration</font></i></div>
<div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils migrate_data)</font></i></div><div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3208, in _post_live_migration</font></i></div>
<div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils migration)</font></i></div><div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line 664, in network_migrate_instance_start</font></i></div>
<div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils migration)</font></i></div><div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 415, in network_migrate_instance_start</font></i></div>
<div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils return self.call(context, msg, version='1.41')</font></i></div><div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line 80, in call</font></i></div>
<div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils return rpc.call(context, self._get_topic(topic), msg, timeout)</font></i></div><div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", line 140, in call</font></i></div>
<div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils return _get_impl().call(CONF, context, topic, msg, timeout)</font></i></div><div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 798, in call</font></i></div>
<div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils rpc_amqp.get_connection_pool(conf, Connection))</font></i></div><div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 612, in call</font></i></div>
<div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils rv = list(rv)</font></i></div><div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 554, in __iter__</font></i></div>
<div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils self.done()</font></i></div><div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__</font></i></div>
<div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils self.gen.next()</font></i></div><div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 551, in __iter__</font></i></div>
<div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils self._iterator.next()</font></i></div><div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 648, in iterconsume</font></i></div>
<div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils yield self.ensure(_error_callback, _consume)</font></i></div><div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 566, in ensure</font></i></div>
<div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils error_callback(e)</font></i></div><div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 629, in _error_callback</font></i></div>
<div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils raise rpc_common.Timeout()</font></i></div><div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils Timeout: Timeout while waiting on RPC response.</font></i></div>
<div><i><font face="courier new, monospace">2013-09-02 15:35:45.790 4601 TRACE nova.utils</font></i></div></div><div><br></div><div><br></div><div>-And the VM state never changes back to ACTIVE from MIGRATING:</div><div><br>
</div><div><br></div><div><div><font face="courier new, monospace"><i>root@cebolla:~/tool# nova --os-username=noc-admin --os-tenant-name=noc --os-password=XXXXX --os-auth-url <a href="http://172.19.136.1:35357/v2.0">http://172.19.136.1:35357/v2.0</a> show de2bcbed-f7b6-40cd-89ca-acf6fe2f2d09</i></font></div>
<div><font face="courier new, monospace"><i>+-------------------------------------+-----------------------------------------------------------+</i></font></div><div><font face="courier new, monospace"><i>| Property | Value |</i></font></div>
<div><font face="courier new, monospace"><i>+-------------------------------------+-----------------------------------------------------------+</i></font></div><div><font face="courier new, monospace"><i>| status | MIGRATING |</i></font></div>
<div><font face="courier new, monospace"><i>| updated | 2013-09-02T15:33:54Z |</i></font></div><div><font face="courier new, monospace"><i>| OS-EXT-STS:task_state | migrating |</i></font></div>
<div><font face="courier new, monospace"><i>| OS-EXT-SRV-ATTR:host | acelga |</i></font></div><div><font face="courier new, monospace"><i>| key_name | None |</i></font></div>
<div><font face="courier new, monospace"><i>| image | Ubuntu 12.04.2 LTS (1359ca8d-23a2-40e8-940f-d90b3e68bb39) |</i></font></div><div><font face="courier new, monospace"><i>| vlan1 network | 172.16.16.175 |</i></font></div>
<div><font face="courier new, monospace"><i>| hostId | 81be94870821e17e327d92e9c80548ffcdd37d24054a235116669f53 |</i></font></div><div><font face="courier new, monospace"><i>| OS-EXT-STS:vm_state | active |</i></font></div>
<div><font face="courier new, monospace"><i>| OS-EXT-SRV-ATTR:instance_name | instance-00000022 |</i></font></div><div><font face="courier new, monospace"><i>| OS-EXT-SRV-ATTR:hypervisor_hostname | <a href="http://acelga.psi.unc.edu.ar">acelga.psi.unc.edu.ar</a> |</i></font></div>
<div><font face="courier new, monospace"><i>| flavor | m1.tiny (1) |</i></font></div><div><font face="courier new, monospace"><i>| id | de2bcbed-f7b6-40cd-89ca-acf6fe2f2d09 |</i></font></div>
<div><font face="courier new, monospace"><i>| security_groups | [{u'name': u'default'}] |</i></font></div><div><font face="courier new, monospace"><i>| user_id | 20390b639d4449c18926dca5e038ec5e |</i></font></div>
<div><font face="courier new, monospace"><i>| name | p9 |</i></font></div><div><font face="courier new, monospace"><i>| created | 2013-09-02T15:27:06Z |</i></font></div>
<div><font face="courier new, monospace"><i>| tenant_id | d1e3aae242f14c488d2225dcbf1e96d6 |</i></font></div><div><font face="courier new, monospace"><i>| OS-DCF:diskConfig | MANUAL |</i></font></div>
<div><font face="courier new, monospace"><i>| metadata | {} |</i></font></div><div><font face="courier new, monospace"><i>| accessIPv4 | |</i></font></div>
<div><font face="courier new, monospace"><i>| accessIPv6 | |</i></font></div><div><font face="courier new, monospace"><i>| OS-EXT-STS:power_state | 1 |</i></font></div>
<div><font face="courier new, monospace"><i>| OS-EXT-AZ:availability_zone | nova |</i></font></div><div><font face="courier new, monospace"><i>| config_drive | |</i></font></div>
<div><font face="courier new, monospace"><i>+-------------------------------------+-----------------------------------------------------------+</i></font></div><div><font face="courier new, monospace"><i>root@cebolla:~/tool#</i></font></div>
</div><div><br></div><div><br></div><div>Funny fact:</div><div>-The vm still answer ping after migration, so i think this is good.</div><div><br></div><div>Any ideas about this problem? At first i thought it could be related to a connection problem between the nodes, but the VM migrates completly in hipervisor level somehow there is some "instance've been migrated ACK" missing.</div>
<div><br></div><div><br></div>-- <br><div dir="ltr">Pavlik Salles Juan José</div>
</div></div>