[Openstack] Cinder volume live migration issue. Pure Storage Cinder-Libvirt
Remo Mattei
Remo at italy1.com
Wed May 30 02:52:00 UTC 2018
We have implemented PURE in the OOO deployment I have, I am going to push Production next week or so with Dual PURE, their docs had some configuration that was not correct, and I am not sure if you follow that or not. I have in fact created a pure Ansible playbook to deploy PURE Flash Array. I can share the points I have setup and you can compare them and see if any of them matches..
Remo
> On May 29, 2018, at 7:05 PM, Steven D. Searles <SSearles at zimcom.net> wrote:
>
> Nothing special, Purity 4.1.5. These are not brand new array’s we have had them for a while. (FAS450) 8gb FC switches (Force 10) All hosts zoned, initiators manually created in the pure or configured by cinder does not seem to matter. The weird thing is everything seems to be working fine except the migration. Provisioning a new instance on the pure no problem, move a volume when not attached and boot on the pure, no probem. Migrate from the pure to the compellent, no problem. If you need more details on the Pure config let me know.
>
> -Steve
>
>
> From: Remo Mattei [mailto:Remo at italy1.com]
> Sent: Tuesday, May 29, 2018 9:55 PM
> To: Steven D. Searles <SSearles at zimcom.net>
> Cc: openstack at lists.openstack.org
> Subject: Re: [Openstack] Cinder volume live migration issue. Pure Storage Cinder-Libvirt
>
> What’s the config on PURE?
>
>
> On May 29, 2018, at 12:03 PM, Steven D. Searles <SSearles at zimcom.net <mailto:SSearles at zimcom.net>> wrote:
>
> Hello everyone, I am seeing a strange issue with cinder block live migration and libvirt and looking for some assistance.
>
> Environment: Openstack Pike
> OS: Ubuntu 16.04LTS
> Cinder FC Driver: Pure Storage
> Cinder FC Driver: Dell Compellent
> libvirtd (libvirt) 3.6.0
> Cinder-volume 2:11.1.0-0ubuntu1~cloud0
>
> I am trying to migrate some volumes from our older Dell Compellent SC storage arrays to new Pure Storage arrays. I can create new volumes on both array’s via cinder. I can create a volume and boot it on the Pure AFA’s. I can live migrate volumes from the Pure array to the Compellent array but not from the Compellent to the Pure. The volume is created and data copied to it, it is then deleted by Cinder after the failure is logged. I receive the following error when going from the Compellent to the Pure Array. I can migrate in both directions if the volume is not attached to an instance. Any ideas?
>
> 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver [req-8ab3bc34-d810-4584-8f31-4dd611ed7b98 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Failure rebasing volume /dev/disk/by-path/pci-0000:03:00.0-fc-0x524a937cddfa5902-lun-2 on vda.: libvirtError: Requested operation is not valid: pivot of disk 'vda' requires an active copy job
> 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver Traceback (most recent call last):
> 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1345, in _swap_volume
> 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver dev.abort_job(pivot=True)
> 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 751, in abort_job
> 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver self._guest._domain.blockJobAbort(self._disk, flags=flags)
> 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
> 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver result = proxy_call(self._autowrap, f, *args, **kwargs)
> 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
> 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver rv = execute(f, *args, **kwargs)
> 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
> 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver six.reraise(c, e, tb)
> 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
> 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver rv = meth(*args, **kwargs)
> 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/libvirt.py", line 766, in blockJobAbort
> 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver if ret == -1: raise libvirtError ('virDomainBlockJobAbort() failed', dom=self)
> 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver libvirtError: Requested operation is not valid: pivot of disk 'vda' requires an active copy job
> 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver
> 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [req-8ab3bc34-d810-4584-8f31-4dd611ed7b98 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] Failed to swap volume bb59023e-d463-44e9-8b1a-a9af495d3d4f for ecc059d0-79f9-402e-aea8-8c99004f221d: VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid: pivot of disk 'vda' requires an active copy job
> 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] Traceback (most recent call last):
> 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5114, in _swap_volume
> 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] mountpoint, resize_to)
> 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1408, in swap_volume
> 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] instance)
> 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
> 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] self.force_reraise()
> 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
> 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] six.reraise(self.type_, self.value, self.tb)
> 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1404, in swap_volume
> 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] self._swap_volume(guest, disk_dev, conf, resize_to)
> 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1351, in _swap_volume
> 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] raise exception.VolumeRebaseFailed(reason=six.text_type(exc))
> 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid: pivot of disk 'vda' requires an active copy job
> 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29]
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server [req-8ab3bc34-d810-4584-8f31-4dd611ed7b98 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Exception during message handling: VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid: pivot of disk 'vda' requires an active copy job
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 76, in wrapped
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server self.force_reraise()
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 67, in wrapped
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 218, in decorated_function
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server self.force_reraise()
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 206, in decorated_function
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5255, in swap_volume
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server is_cinder_migration)
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5154, in _swap_volume
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server context, new_attachment_id)
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server self.force_reraise()
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5114, in _swap_volume
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server mountpoint, resize_to)
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1408, in swap_volume
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server instance)
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server self.force_reraise()
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1404, in swap_volume
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server self._swap_volume(guest, disk_dev, conf, resize_to)
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1351, in _swap_volume
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server raise exception.VolumeRebaseFailed(reason=six.text_type(exc))
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid: pivot of disk 'vda' requires an active copy job
> 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server
>
>
> Any help would be greatly appreciated.
>
> Thanks!
>
> — Steve
>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack>
> Post to : openstack at lists.openstack.org <mailto:openstack at lists.openstack.org>
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20180529/5ffed62f/attachment-0001.html>
More information about the Openstack
mailing list