[Openstack-operators] Migrating instances in grizzly

Emilien Macchi emilien.macchi at enovance.com
Thu Sep 5 13:02:27 UTC 2013


Hi,

We have the same issue here with Grizzly 2013.1.2 / Ubuntu 12.04 / 
libvirt 1.0.2.

Which release are you running ?

Emilien Macchi
----------------------------------------------------
# OpenStack Engineer
// eNovance Inc.              http://enovance.com
// ✉ emilien at enovance.com     ☎ +33 (0)1 49 70 99 80
// 10 rue de la Victoire 75009 Paris

On Tue 03 Sep 2013 02:44:09 AM CEST, Juan José Pavlik Salles wrote:
> I've also found this in nova-conductor.log:
>
> 2013-09-02 15:35:27.208 DEBUG nova.openstack.common.rpc.common
> [req-e0473533-89af-4ff5-b6fa-4b0b6eb50a6d
> 31020076174943bdb7486c330a298d93 d1e3aae242f14c488d2225dc
> bf1e96d6] Timed out waiting for RPC response: timed out
> _error_callback
> /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py:628
> 2013-09-02 15:35:27.222 ERROR nova.openstack.common.rpc.amqp
> [req-e0473533-89af-4ff5-b6fa-4b0b6eb50a6d
> 31020076174943bdb7486c330a298d93 d1e3aae242f14c488d2225dcbf
> 1e96d6] Exception during message handling
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> Traceback (most recent call last):
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
> line 430, in _proce
> ss_data
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> rval = self.proxy.dispatch(ctxt, version, method, **args)
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
> line 133, in
> dispatch
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> return getattr(proxyobj, method)(ctxt, **kwargs)
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py",
> line 399, in network_migrat
> e_instance_start
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> self.network_api.migrate_instance_start(context, instance, migration)
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 89,
> in wrapped
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> return func(self, context, *args, **kwargs)
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 501,
> in migrate_instance_sta
> rt
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> self.network_rpcapi.migrate_instance_start(context, **args)
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line
> 333, in migrate_instance_
> start
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> version='1.2')
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py",
> line 80, in call
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> return rpc.call(context, self._get_topic(topic), msg, timeout)
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py",
> line 140, in ca
> ll
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> return _get_impl().call(CONF, context, topic, msg, timeout)
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
> line 798, in
> call
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> rpc_amqp.get_connection_pool(conf, Connection))
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
> line 612, in call
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> rv = list(rv)
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
> line 554, in __iter__
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> self.done()
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> self.gen.next()
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
> line 551, in __iter__
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> self._iterator.next()
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
> line 648, in iterconsume
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> yield self.ensure(_error_callback, _consume)
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
> line 566, in ensure
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> error_callback(e)
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
> line 629, in _error_callback
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> raise rpc_common.Timeout()
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> Timeout: Timeout while waiting on RPC response.
> 2013-09-02 15:35:27.222 1363 TRACE nova.openstack.common.rpc.amqp
> 2013-09-02 15:35:27.237 ERROR nova.openstack.common.rpc.common
> [req-e0473533-89af-4ff5-b6fa-4b0b6eb50a6d
> 31020076174943bdb7486c330a298d93 d1e3aae242f14c488d2225dcbf1e96d6]
> Returning exception Timeout while waiting on RPC response. to caller
>
> Does anybody know all the steps that take to live-migrate an instance
> ?? It seems to be stopping inside the network_migrate_instance_start
> function, really no clue at all...
>
>
> 2013/9/2 Juan José Pavlik Salles <jjpavlik at gmail.com
> <mailto:jjpavlik at gmail.com>>
>
>     Hi guys, last friday i started testing live-migration in my
>     grizzly cloud with shared storage (gfs2) but i run into a problem,
>     a little weird:
>
>     This is the status before migrating:
>
>     -I've p9 instances also called instance-00000022 running on
>     "acelga" compute node.
>
>     /root at acelga:~/tools# virsh list/
>     / Id    Name                           State/
>     /----------------------------------------------------/
>     / 6     instance-00000022              running/
>     /
>     /
>     /root at acelga:~/tools# /
>     /
>     /
>     /
>     /
>     /root at cebolla:~/tool# virsh list/
>     / Id    Nombre                         Estado/
>     /----------------------------------------------------/
>     /
>     /
>     /root at cebolla:~/tool# /
>
>     -Here you can see all the info about the instance
>
>     /root at cebolla:~/tool# nova --os-username=noc-admin
>     --os-tenant-name=noc --os-password=XXXXXXX --os-auth-url
>     http://172.19.136.1:35357/v2.0 show
>     de2bcbed-f7b6-40cd-89ca-acf6fe2f2d09/
>     /+-------------------------------------+-----------------------------------------------------------+/
>     /| Property                            | Value
>                                     |/
>     /+-------------------------------------+-----------------------------------------------------------+/
>     /| status                              | ACTIVE
>                                      |/
>     /| updated                             | 2013-09-02T15:27:39Z
>                                      |/
>     /| OS-EXT-STS:task_state               | None
>                                      |/
>     /| OS-EXT-SRV-ATTR:host                | acelga
>                                      |/
>     /| key_name                            | None
>                                      |/
>     /| image                               | Ubuntu 12.04.2 LTS
>     (1359ca8d-23a2-40e8-940f-d90b3e68bb39) |/
>     /| vlan1 network                       | 172.16.16.175
>                                     |/
>     /| hostId                              |
>     81be94870821e17e327d92e9c80548ffcdd37d24054a235116669f53  |/
>     /| OS-EXT-STS:vm_state                 | active
>                                      |/
>     /| OS-EXT-SRV-ATTR:instance_name       | instance-00000022
>                                     |/
>     /| OS-EXT-SRV-ATTR:hypervisor_hostname | acelga.psi.unc.edu.ar
>     <http://acelga.psi.unc.edu.ar>                                     |/
>     /| flavor                              | m1.tiny (1)
>                                     |/
>     /| id                                  |
>     de2bcbed-f7b6-40cd-89ca-acf6fe2f2d09                      |/
>     /| security_groups                     | [{u'name': u'default'}]
>                                     |/
>     /| user_id                             |
>     20390b639d4449c18926dca5e038ec5e                          |/
>     /| name                                | p9
>                                      |/
>     /| created                             | 2013-09-02T15:27:06Z
>                                      |/
>     /| tenant_id                           |
>     d1e3aae242f14c488d2225dcbf1e96d6                          |/
>     /| OS-DCF:diskConfig                   | MANUAL
>                                      |/
>     /| metadata                            | {}
>                                      |/
>     /| accessIPv4                          |
>                                     |/
>     /| accessIPv6                          |
>                                     |/
>     /| progress                            | 0
>                                     |/
>     /| OS-EXT-STS:power_state              | 1
>                                     |/
>     /| OS-EXT-AZ:availability_zone         | nova
>                                      |/
>     /| config_drive                        |
>                                     |/
>     /+-------------------------------------+-----------------------------------------------------------+/
>     /root at cebolla:~/tool#/
>
>     -So i try to move it to the other node "cebolla"
>
>     /root at acelga:~/tools# nova --os-username=noc-admin
>     --os-tenant-name=noc --os-password=HjZ5V9yj --os-auth-url
>     http://172.19.136.1:35357/v2.0 live-migration
>     de2bcbed-f7b6-40cd-89ca-acf6fe2f2d09 cebolla
>     /
>     /root at acelga:~/tools# virsh list/
>     / Id    Name                           State/
>     /----------------------------------------------------/
>     /
>     /
>     /root at acelga:~/tools#/
>
>     No error messages at all on "acelga" compute node so far. If i
>     check the other node i can see the instance've been migrated
>
>     /root at cebolla:~/tool# virsh list/
>     / Id    Nombre                         Estado/
>     /----------------------------------------------------/
>     / 11    instance-00000022              ejecutando/
>     /
>     /
>     /root at cebolla:~/tool#/
>
>
>     -BUT... after a few seconds i get this on "acelga"'s nova-compute.log
>
>
>     /2013-09-02 15:35:45.784 4601 DEBUG
>     nova.openstack.common.rpc.common [-] Timed out waiting for RPC
>     response: timed out _error_callback
>     /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py:628/
>     /2013-09-02 15:35:45.790 4601 ERROR nova.utils [-] in fixed
>     duration looping call/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils Traceback (most
>     recent call last):/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils   File
>     "/usr/lib/python2.7/dist-packages/nova/utils.py", line 594, in _inner/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils
>     self.f(*self.args, **self.kw <http://self.kw>)/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils   File
>     "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py",
>     line 3129, in wait_for_live_migration/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils     migrate_data)/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils   File
>     "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>     3208, in _post_live_migration/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils     migration)/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils   File
>     "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line
>     664, in network_migrate_instance_start/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils     migration)/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils   File
>     "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line
>     415, in network_migrate_instance_start/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils     return
>     self.call(context, msg, version='1.41')/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils   File
>     "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py",
>     line 80, in call/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils     return
>     rpc.call(context, self._get_topic(topic), msg, timeout)/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils   File
>     "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py",
>     line 140, in call/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils     return
>     _get_impl().call(CONF, context, topic, msg, timeout)/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils   File
>     "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>     line 798, in call/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils
>     rpc_amqp.get_connection_pool(conf, Connection))/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils   File
>     "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
>     line 612, in call/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils     rv = list(rv)/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils   File
>     "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
>     line 554, in __iter__/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils     self.done()/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils   File
>     "/usr/lib/python2.7/contextlib.py", line 24, in __exit__/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils     self.gen.next()/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils   File
>     "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
>     line 551, in __iter__/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils
>     self._iterator.next()/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils   File
>     "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>     line 648, in iterconsume/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils     yield
>     self.ensure(_error_callback, _consume)/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils   File
>     "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>     line 566, in ensure/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils     error_callback(e)/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils   File
>     "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>     line 629, in _error_callback/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils     raise
>     rpc_common.Timeout()/
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils Timeout: Timeout
>     while waiting on RPC response./
>     /2013-09-02 15:35:45.790 4601 TRACE nova.utils/
>
>
>     -And the VM state never changes back to ACTIVE from MIGRATING:
>
>
>     /root at cebolla:~/tool# nova --os-username=noc-admin
>     --os-tenant-name=noc --os-password=XXXXX --os-auth-url
>     http://172.19.136.1:35357/v2.0 show
>     de2bcbed-f7b6-40cd-89ca-acf6fe2f2d09/
>     /+-------------------------------------+-----------------------------------------------------------+/
>     /| Property                            | Value
>                                     |/
>     /+-------------------------------------+-----------------------------------------------------------+/
>     /| status                              | MIGRATING
>                                     |/
>     /| updated                             | 2013-09-02T15:33:54Z
>                                      |/
>     /| OS-EXT-STS:task_state               | migrating
>                                     |/
>     /| OS-EXT-SRV-ATTR:host                | acelga
>                                      |/
>     /| key_name                            | None
>                                      |/
>     /| image                               | Ubuntu 12.04.2 LTS
>     (1359ca8d-23a2-40e8-940f-d90b3e68bb39) |/
>     /| vlan1 network                       | 172.16.16.175
>                                     |/
>     /| hostId                              |
>     81be94870821e17e327d92e9c80548ffcdd37d24054a235116669f53  |/
>     /| OS-EXT-STS:vm_state                 | active
>                                      |/
>     /| OS-EXT-SRV-ATTR:instance_name       | instance-00000022
>                                     |/
>     /| OS-EXT-SRV-ATTR:hypervisor_hostname | acelga.psi.unc.edu.ar
>     <http://acelga.psi.unc.edu.ar>                                     |/
>     /| flavor                              | m1.tiny (1)
>                                     |/
>     /| id                                  |
>     de2bcbed-f7b6-40cd-89ca-acf6fe2f2d09                      |/
>     /| security_groups                     | [{u'name': u'default'}]
>                                     |/
>     /| user_id                             |
>     20390b639d4449c18926dca5e038ec5e                          |/
>     /| name                                | p9
>                                      |/
>     /| created                             | 2013-09-02T15:27:06Z
>                                      |/
>     /| tenant_id                           |
>     d1e3aae242f14c488d2225dcbf1e96d6                          |/
>     /| OS-DCF:diskConfig                   | MANUAL
>                                      |/
>     /| metadata                            | {}
>                                      |/
>     /| accessIPv4                          |
>                                     |/
>     /| accessIPv6                          |
>                                     |/
>     /| OS-EXT-STS:power_state              | 1
>                                     |/
>     /| OS-EXT-AZ:availability_zone         | nova
>                                      |/
>     /| config_drive                        |
>                                     |/
>     /+-------------------------------------+-----------------------------------------------------------+/
>     /root at cebolla:~/tool#/
>
>
>     Funny fact:
>     -The vm still answer ping after migration, so i think this is good.
>
>     Any ideas about this problem? At first i thought it could be
>     related to a connection problem between the nodes, but the VM
>     migrates completly in hipervisor level somehow there is some
>     "instance've been migrated ACK" missing.
>
>
>     --
>     Pavlik Salles Juan José
>
>
>
>
> --
> Pavlik Salles Juan José
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 553 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20130905/a5a4da9b/attachment.pgp>


More information about the OpenStack-operators mailing list