<div dir="ltr"><div>Do not worry, I know queens is very old. <br></div><div>I am going to move volumes with another procedure.</div><div>Ignazio<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno gio 10 mar 2022 alle ore 13:58 Gorka Eguileor <<a href="mailto:geguileo@redhat.com">geguileo@redhat.com</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 07/03, Ignazio Cassano wrote:<br>
> Hello again I found the error but it is in cinder api log:<br>
> 2022-03-07 15:42:57.330 156916 INFO cinder.api.openstack.wsgi<br>
> [req-01eb2613-4aad-4809-ac91-037de3148fbb 5134ed3b93284af5ba2f05d7361edf53<br>
> 0e760ccde5d24af5a571de40220fbf80 - default default]<br>
> <a href="http://10.138.144.16:8776/v3/0e760ccde5d24af5a571de40220fbf80/attachments/04ae01f6-e164-40cb-8d31-63f78e69c8db" rel="noreferrer" target="_blank">http://10.138.144.16:8776/v3/0e760ccde5d24af5a571de40220fbf80/attachments/04ae01f6-e164-40cb-8d31-63f78e69c8db</a><br>
> returned with HTTP 404<br>
><br>
> and in nova compute log:<br>
> 15:42:57.502 280431 ERROR oslo_messaging.rpc.server<br>
> [req-dfeac1f0-870b-4aef-af40-cda52f4552e0 5134ed3b93284af5ba2f05d7361edf53<br>
> 0e760ccde5d24af5a571de40220fbf80 - default default] Exception during<br>
> message handling: VolumeAttachmentNotFound: Volume attachment<br>
> 04ae01f6-e164-40cb-8d31-63f78e69c8db could not be found.<br>
><br>
<br>
Hi,<br>
<br>
Retyping with migration of attached volumes is a complex flow, so doing<br>
the RCA usually requires full logs of ALL involved services:<br>
<br>
- All nova-api, cinder-api, cinder-scheduler services<br>
- nova-cpu of the host with the instance<br>
- Cinder-volume service<br>
<br>
For example in the logs you gave above we don't even know if it was a<br>
GET or PUT operation and what was nova trying to do, and we also don't<br>
have context, so we cannot tell at which stage of the whole process this<br>
happened.<br>
<br>
If I remember correctly the high level steps are something like:<br>
<br>
- Cinder-volume receives request<br>
- Creates new volume<br>
- Calls nova to do the volume swap<br>
- Nova attaches new volume with os-brick<br>
- Nova asks the hypervisor to do the swap<br>
- Nova disconnects the original volume<br>
- Nova calls cinder telling it that the swap is completed<br>
- Cinder detaches the original volume and makes DB changes<br>
<br>
Since you are running a 4 years old code it's not easy to see the code<br>
differences to see if there was a bug that has been fixed in a later<br>
release.<br>
<br>
Sorry for not being able to provide actual help. :-(<br>
<br>
Cheers,<br>
Gorka.<br>
<br>
<br>
> Il giorno lun 7 mar 2022 alle ore 17:44 Ignazio Cassano <<br>
> <a href="mailto:ignaziocassano@gmail.com" target="_blank">ignaziocassano@gmail.com</a>> ha scritto:<br>
><br>
> > Hello, volume.log does not report errors:<br>
> > 2022-03-07 15:41:44.435 159559 INFO cinder.volume.manager<br>
> > [req-68444973-24eb-4b21-95b2-626c9b2e2aa0 5134ed3b93284af5ba2f05d7361edf53<br>
> > 0e760ccde5d24af5a571de40220fbf80 - default default] Created volume<br>
> > successfully.<br>
> > 2022-03-07 15:41:45.358 159099 INFO cinder.volume.manager<br>
> > [req-68444973-24eb-4b21-95b2-626c9b2e2aa0 5134ed3b93284af5ba2f05d7361edf53<br>
> > 0e760ccde5d24af5a571de40220fbf80 - default default] Migrate volume<br>
> > completed successfully.<br>
> > 2022-03-07 15:41:48.970 159559 INFO cinder.volume.manager<br>
> > [req-72ca9242-25e9-4bb0-9bd6-9857f42805ee 5134ed3b93284af5ba2f05d7361edf53<br>
> > 0e760ccde5d24af5a571de40220fbf80 - default default] attachment_update<br>
> > complete<br>
> ><br>
> > Il giorno lun 7 mar 2022 alle ore 17:25 Eugen Block <<a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a>> ha<br>
> > scritto:<br>
> ><br>
> >> Hi,<br>
> >><br>
> >> since you're retyping a volume, what do the cinder logs reveal?<br>
> >><br>
> >><br>
> >> Zitat von Ignazio Cassano <<a href="mailto:ignaziocassano@gmail.com" target="_blank">ignaziocassano@gmail.com</a>>:<br>
> >><br>
> >> > Hello, sometimes the volume retype from a netapp nfs storage to another<br>
> >> > netapp nfs storage does not work.<br>
> >> > I do not know the reason but I think something is going wrong in nova:<br>
> >> ><br>
> >> > /var/log/nova/nova-compute.log:2022-02-02 14:23:39.992 194288 ERROR root<br>
> >> > [req-ebbf749f-359f-4282-94c5-0913cffe97a3<br>
> >> 66adb965bef64eaaab2af93ade87e2ca<br>
> >> > 68feefdf7c5847388534e9214dc13366 - default default] Original exception<br>
> >> > being dropped: ['Traceback (most recent call last):\n', ' File<br>
> >> > "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5655,<br>
> >> in<br>
> >> > _swap_volume\n mountpoint, resize_to)\n', ' File<br>
> >> > "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line<br>
> >> 1647,<br>
> >> > in swap_volume\n self._disconnect_volume(context,<br>
> >> new_connection_info,<br>
> >> > instance)\n', ' File<br>
> >> > "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in<br>
> >> > __exit__\n self.force_reraise()\n', ' File<br>
> >> > "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in<br>
> >> > force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', '<br>
> >> File<br>
> >> > "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line<br>
> >> 1644,<br>
> >> > in swap_volume\n self._swap_volume(guest, disk_dev, conf,<br>
> >> resize_to)\n',<br>
> >> > ' File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",<br>
> >> > line 1593, in _swap_volume\n raise<br>
> >> > exception.VolumeRebaseFailed(reason=six.text_type(exc))\n',<br>
> >> > "VolumeRebaseFailed: Volume rebase failed: Requested operation is not<br>
> >> > valid: pivot of disk 'vda' requires an active copy job\n"]:<br>
> >> > VolumeAttachmentNotFound: Volume attachment<br>
> >> > 2cd820e0-85e8-498d-a62a-800260d0cf31 could not be found<br>
> >> ><br>
> >> ><br>
> >> > Must I try to change some parameters in nova.conf on compute nodes about<br>
> >> > volume attachment ?<br>
> >> ><br>
> >> > Thanks<br>
> >> > Ignazio<br>
> >><br>
> >><br>
> >><br>
> >><br>
> >><br>
<br>
</blockquote></div>