<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
Hello everyone, I am seeing a strange issue with cinder block live migration and libvirt and looking for some assistance. </div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
Environment: Openstack Pike </div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
OS: Ubuntu 16.04LTS</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
Cinder FC Driver: Pure Storage</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
Cinder FC Driver: Dell Compellent </div>
<div><font face="Calibri,sans-serif">libvirtd (libvirt) 3.6.0</font></div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
Cinder-volume 2:11.1.0-0ubuntu1~cloud0</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
I am trying to migrate some volumes from our older Dell Compellent SC storage arrays to new Pure Storage arrays. I can create new volumes on both array’s via cinder. I can create a volume and boot it on the Pure AFA’s. I can live migrate volumes from the
Pure array to the Compellent array but not from the Compellent to the Pure. The volume is created and data copied to it, it is then deleted by Cinder after the failure is logged. I receive the following error when going from the Compellent to the Pure Array.
I can migrate in both directions if the volume is not attached to an instance. Any ideas? </div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<div>2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver [req-8ab3bc34-d810-4584-8f31-4dd611ed7b98 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Failure rebasing volume /dev/disk/by-path/pci-0000:03:00.0-fc-0x524a937cddfa5902-lun-2
on vda.: libvirtError: Requested operation is not valid: pivot of disk 'vda' requires an active copy job</div>
<div>2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver Traceback (most recent call last):</div>
<div>2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1345, in _swap_volume</div>
<div>2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver dev.abort_job(pivot=True)</div>
<div>2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 751, in abort_job</div>
<div>2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver self._guest._domain.blockJobAbort(self._disk, flags=flags)</div>
<div>2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit</div>
<div>2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver result = proxy_call(self._autowrap, f, *args, **kwargs)</div>
<div>2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call</div>
<div>2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver rv = execute(f, *args, **kwargs)</div>
<div>2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute</div>
<div>2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver six.reraise(c, e, tb)</div>
<div>2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker</div>
<div>2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver rv = meth(*args, **kwargs)</div>
<div>2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/libvirt.py", line 766, in blockJobAbort</div>
<div>2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver if ret == -1: raise libvirtError ('virDomainBlockJobAbort() failed', dom=self)</div>
<div>2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver libvirtError: Requested operation is not valid: pivot of disk 'vda' requires an active copy job</div>
<div>2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver </div>
<div>2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [req-8ab3bc34-d810-4584-8f31-4dd611ed7b98 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] Failed to swap volume
bb59023e-d463-44e9-8b1a-a9af495d3d4f for ecc059d0-79f9-402e-aea8-8c99004f221d: VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid: pivot of disk 'vda' requires an active copy job</div>
<div>2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] Traceback (most recent call last):</div>
<div>2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5114, in _swap_volume</div>
<div>2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] mountpoint, resize_to)</div>
<div>2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1408, in swap_volume</div>
<div>2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] instance)</div>
<div>2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__</div>
<div>2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] self.force_reraise()</div>
<div>2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise</div>
<div>2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] six.reraise(self.type_, self.value, self.tb)</div>
<div>2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1404, in swap_volume</div>
<div>2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] self._swap_volume(guest, disk_dev, conf, resize_to)</div>
<div>2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1351, in _swap_volume</div>
<div>2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] raise exception.VolumeRebaseFailed(reason=six.text_type(exc))</div>
<div>2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid: pivot of disk 'vda' requires an active copy job</div>
<div>2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] </div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server [req-8ab3bc34-d810-4584-8f31-4dd611ed7b98 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Exception during message handling: VolumeRebaseFailed: Volume rebase
failed: Requested operation is not valid: pivot of disk 'vda' requires an active copy job</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server Traceback (most recent call last):</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 76, in wrapped</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server self.force_reraise()</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 67, in wrapped</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 218, in decorated_function</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server self.force_reraise()</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 206, in decorated_function</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5255, in swap_volume</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server is_cinder_migration)</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5154, in _swap_volume</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server context, new_attachment_id)</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server self.force_reraise()</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5114, in _swap_volume</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server mountpoint, resize_to)</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1408, in swap_volume</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server instance)</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server self.force_reraise()</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1404, in swap_volume</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server self._swap_volume(guest, disk_dev, conf, resize_to)</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1351, in _swap_volume</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server raise exception.VolumeRebaseFailed(reason=six.text_type(exc))</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid: pivot of disk 'vda' requires an active copy job</div>
<div>2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server </div>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
Any help would be greatly appreciated. </div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
Thanks!</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<div id="">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Cambria;
panose-1:2 4 5 3 5 4 6 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman";}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.EmailStyle19
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal-compose;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026"/>
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1"/>
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal" style="text-autospace:none">— Steve </p>
<p class="MsoNormal" style="text-autospace:none"><br>
</p>
<p class="MsoNormal" style="text-autospace:none"><br>
</p>
</div>
</div>
</div>
</body>
</html>