[Openstack] Cinder volume live migration issue. Pure Storage Cinder-Libvirt

Steven D. Searles SSearles at zimcom.net
Tue May 29 19:03:42 UTC 2018


Hello everyone, I am seeing a strange issue with cinder block live migration and libvirt and looking for some assistance.

Environment: Openstack Pike
OS: Ubuntu 16.04LTS
Cinder FC Driver: Pure Storage
Cinder FC Driver: Dell Compellent
libvirtd (libvirt) 3.6.0
Cinder-volume 2:11.1.0-0ubuntu1~cloud0

I am trying to migrate some volumes from our older Dell Compellent SC storage arrays to new Pure Storage arrays.  I can create new volumes on both array’s via cinder. I can create a volume and boot it on the Pure AFA’s.  I can live migrate volumes from the Pure array to the Compellent array but not from the Compellent to the Pure. The volume is created and data copied to it, it is then deleted by Cinder after the failure is logged.  I receive the following error when going from the Compellent to the Pure Array.  I can migrate in both directions if the volume is not attached to an instance.  Any ideas?

2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver [req-8ab3bc34-d810-4584-8f31-4dd611ed7b98 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Failure rebasing volume /dev/disk/by-path/pci-0000:03:00.0-fc-0x524a937cddfa5902-lun-2 on vda.: libvirtError: Requested operation is not valid: pivot of disk 'vda' requires an active copy job
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver Traceback (most recent call last):
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1345, in _swap_volume
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver     dev.abort_job(pivot=True)
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 751, in abort_job
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver     self._guest._domain.blockJobAbort(self._disk, flags=flags)
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver     result = proxy_call(self._autowrap, f, *args, **kwargs)
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver     rv = execute(f, *args, **kwargs)
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver     six.reraise(c, e, tb)
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver     rv = meth(*args, **kwargs)
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 766, in blockJobAbort
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver     if ret == -1: raise libvirtError ('virDomainBlockJobAbort() failed', dom=self)
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver libvirtError: Requested operation is not valid: pivot of disk 'vda' requires an active copy job
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [req-8ab3bc34-d810-4584-8f31-4dd611ed7b98 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] Failed to swap volume bb59023e-d463-44e9-8b1a-a9af495d3d4f for ecc059d0-79f9-402e-aea8-8c99004f221d: VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid: pivot of disk 'vda' requires an active copy job
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] Traceback (most recent call last):
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5114, in _swap_volume
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29]     mountpoint, resize_to)
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1408, in swap_volume
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29]     instance)
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29]     self.force_reraise()
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29]     six.reraise(self.type_, self.value, self.tb)
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1404, in swap_volume
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29]     self._swap_volume(guest, disk_dev, conf, resize_to)
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1351, in _swap_volume
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29]     raise exception.VolumeRebaseFailed(reason=six.text_type(exc))
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid: pivot of disk 'vda' requires an active copy job
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29]
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server [req-8ab3bc34-d810-4584-8f31-4dd611ed7b98 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Exception during message handling: VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid: pivot of disk 'vda' requires an active copy job
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 76, in wrapped
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 67, in wrapped
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 218, in decorated_function
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 206, in decorated_function
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5255, in swap_volume
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     is_cinder_migration)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5154, in _swap_volume
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     context, new_attachment_id)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5114, in _swap_volume
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     mountpoint, resize_to)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1408, in swap_volume
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     instance)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1404, in swap_volume
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     self._swap_volume(guest, disk_dev, conf, resize_to)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1351, in _swap_volume
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server     raise exception.VolumeRebaseFailed(reason=six.text_type(exc))
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid: pivot of disk 'vda' requires an active copy job
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server


Any help would be greatly appreciated.

Thanks!

— Steve


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20180529/f83fa431/attachment.html>


More information about the Openstack mailing list