Open Stack

Hi All,

I'm just starting to poke around at live-migration (Ubuntu 12.04,
Havana, KVM), and I'm seeing what seems like a bug but want to see if
it's an obvious problem with what I'm doing before I report it as
such.

using:
 nova live-migration --block_migrate
6fb68161-f9df-4fbc-b557-22f00c65bc59 new-host


The instance does in fact migrate to new-host.  I can see it in virsh
list and see traffic on it's tap device, disk and console.log exist in
/var/lib/nova/instances/<uuid>, but not libvirt.xml. It is gone from
the old-host as shown by virsh (not even in virsh list --all), though
all the files in /var/lib/nova/instances/<uuid> are still there.

But nova still reports it on old-host and in "MIGRATING" status.

Am I doing anything obviously broken here?

relevant nova.conf settings on old-host:

live_migration_bandwidth=0
live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE
live_migration_retry_count=30
live_migration_uri=qemu+ssh://nova@%s/system?keyfile=/var/lib/nova/.ssh/id_rsa&no_tty=1


There is an RPC  timeout error on old-host nova-compute.log:

2014-02-03 14:52:54.942 29816 ERROR nova.openstack.common.loopingcall
[-] in fixed duration looping call
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
Traceback (most recent call last):
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
 File "/usr/lib/python2.7/dist-packages/nova/openstack/common/loopingcall.py",
line 78, in _inner
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
   self.f(*self.args, **self.kw)
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
 File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py",
line 4066, in wait_for_live_migration
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
   migrate_data)
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
 File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90,
in wrapped
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
   payload)
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
 File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73,
in wrapped
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
   return f(self, context, *args, **kw)
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
 File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
4117, in _post_live_migration
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
   migration)
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
 File "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line
305, in network_migrate_instance_start
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
   migration)
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
 File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py",
line 451, in network_migrate_instance_start
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
   instance=instance_p, migration=migration_p)
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
 File "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 85,
in call
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
   return self._invoke(self.proxy.call, ctxt, method, **kwargs)
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
 File "/usr/lib/python2.7/dist-packages/
nova/rpcclient.py", line 63, in _invoke
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
   return cast_or_call(ctxt, msg, **self.kwargs)
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
 File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py",
line 130, in call
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
   exc.info, real_topic, msg.get('method'))
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
Timeout: Timeout while waiting on RPC response - topic: "conductor",
RPC method: "network_migrate_instance_start" info: "<unknown>"
2014-02-03 14:52:54.942 29816 TRACE nova.openstack.common.loopingcall
2014-02-03 14:52:55.157 29816 WARNING nova.openstack.common.rpc.amqp
[-] No calling threads waiting for msg_id :
3315bf59e8fb4e0fa6df87f5c8964804, message : {u'_unique_id':
u'74600572ae784d1db63c467ab8d87160', u'failure': u'{"args": ["Timeout
while waiting on RPC response - topic: \\"network\\", RPC method:
\\"migrate_instance_start\\" info: \\"<unknown>\\""], "module":
"nova.openstack.common.rpc.common", "kwargs": {"info": "<unknown>",
"topic": "network", "method": "migrate_instance_start"}, "message":
"Timeout while waiting on RPC response - topic: \\"network\\", RPC
method: \\"migrate_instance_start\\" info: \\"<unknown>\\"", "tb":
["Traceback (most recent call last):\\n", "  File
\\"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py\\",
line 461, in _process_data\\n    **args)\\n", "  File
\\"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py\\",
line 172, in dispatch\\n    result = getattr(proxyobj, method)(ctxt,
**kwargs)\\n", "  File
\\"/usr/lib/python2.7/dist-packages/nova/conductor/manager.py\\", line
495, in network_migrate_instance_start\\n
self.network_api.migrate_instance_start(context, instance,
migration)\\n", "  File
\\"/usr/lib/python2.7/dist-packages/nova/network/api.py\\", line 93,
in wrapped\\n    return func(self, context, *args, **kwargs)\\n", "
File \\"/usr/lib/python2.7/dist-packages/nova/network/api.py\\", line
539, in migrate_instance_start\\n
self.network_rpcapi.migrate_instance_start(context, **args)\\n", "
File \\"/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py\\",
line 347, in migrate_instance_start\\n
floating_addresses=floating_addresses)\\n", "  File
\\"/usr/lib/python2.7/dist-packages/nova/rpcclient.py\\", line 85, in
call\\n    return self._invoke(self.proxy.call, ctxt, method,
**kwargs)\\n", "  File
\\"/usr/lib/python2.7/dist-packages/nova/rpcclient.py\\", line 63, in
_invoke\\n    return cast_or_call(ctxt, msg, **self.kwargs)\\n", "
File \\"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py\\",
line 130, in call\\n    exc.info, real_topic,
msg.get(\'method\'))\\n", "Timeout: Timeout while waiting on RPC
response - topic: \\"network\\", RPC method:
\\"migrate_instance_start\\" info: \\"<unknown>\\"\\n"], "class":
"Timeout"}', u'result': None}

Thanks,
-Jon