Open Stack

Hi All,

In the process of a Kilo to Liberty upgrade and cannot live migrate VMs between Kilo and Liberty compute hosts. The Control nodes and a single Compute node have been upgraded to Liberty. We get the following error in the compute nodes log when we attempt to live migrate a instance from the Kilo node to the Liberty node;

Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5314, in live_migration
     block_migration, disk, dest, migrate_data)
   File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 623, in pre_live_migration
     disk=disk, migrate_data=migrate_data)
   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
    retry=self.retry)
   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
    timeout=timeout, retry=retry)
   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
     retry=retry)
   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
     raise result
 RemoteError: Remote error: UnsupportedVersion Endpoint does not support RPC version 3.19. Attempted met
hod: pre_live_migration


Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
     executor_callback))
   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
     executor_callback)
   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
     result = func(ctxt, **new_args)
   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6810, in live_migration
     migrate_data=migrate_data)
   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
     payload)
   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
     six.reraise(self.type_, self.value, self.tb)
   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
     return f(self, context, *args, **kw)
   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 361, in decorated_function
     kwargs['instance'], e, sys.exc_info())
   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
     six.reraise(self.type_, self.value, self.tb)
   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 349, in decorated_function
     return function(self, context, *args, **kwargs)
   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5322, in live_migration
     block_migration, migrate_data)
   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
     payload)
   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
     six.reraise(self.type_, self.value, self.tb)
   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
     return f(self, context, *args, **kw)
   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 361, in decorated_function
     kwargs['instance'], e, sys.exc_info())
   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
     six.reraise(self.type_, self.value, self.tb)
   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 349, in decorated_function
     return function(self, context, *args, **kwargs)
   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5564, in _rollback_live_migration
     context, instance, bdm.volume_id, dest)
   File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 715, in remove_volume_connection
     instance=instance, volume_id=volume_id)
   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
     retry=self.retry)
   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
     timeout=timeout, retry=retry)
   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
     retry=retry)
   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
     raise result
 RemoteError: Remote error: UnsupportedVersion Endpoint does not support RPC version 3.30. Attempted method: remove_volume_connection


Upgrade level has been set to "kilo" on all Nova Control and Compute nodes in /etc/nova/nova.conf;
[upgrade_levels]
compute = kilo

Similar errors are raised during an offline migration or resize. Live migration worked correctly with Liberty Control nodes and Kilo Compute nodes. Instances can be booted and run on both Liberty and Kilo Compute nodes. Any idea what's going on here?

Regards,
Oisin

Oisin O'Malley
Systems Engineer
Iocane Pty Ltd
763 South Road
Black Forest SA 5035

Office:+61 (8) 8413 1010
Fax:+61 (8) 8231 2050
Email:oisin.omalley at iocane.com.au
Web:www.iocane.com.au

Better for business