Do not worry, I know queens is very old. I am going to move volumes with another procedure. Ignazio Il giorno gio 10 mar 2022 alle ore 13:58 Gorka Eguileor <geguileo@redhat.com> ha scritto:
On 07/03, Ignazio Cassano wrote:
Hello again I found the error but it is in cinder api log: 2022-03-07 15:42:57.330 156916 INFO cinder.api.openstack.wsgi [req-01eb2613-4aad-4809-ac91-037de3148fbb 5134ed3b93284af5ba2f05d7361edf53 0e760ccde5d24af5a571de40220fbf80 - default default]
http://10.138.144.16:8776/v3/0e760ccde5d24af5a571de40220fbf80/attachments/04...
returned with HTTP 404
and in nova compute log: 15:42:57.502 280431 ERROR oslo_messaging.rpc.server [req-dfeac1f0-870b-4aef-af40-cda52f4552e0 5134ed3b93284af5ba2f05d7361edf53 0e760ccde5d24af5a571de40220fbf80 - default default] Exception during message handling: VolumeAttachmentNotFound: Volume attachment 04ae01f6-e164-40cb-8d31-63f78e69c8db could not be found.
Hi,
Retyping with migration of attached volumes is a complex flow, so doing the RCA usually requires full logs of ALL involved services:
- All nova-api, cinder-api, cinder-scheduler services - nova-cpu of the host with the instance - Cinder-volume service
For example in the logs you gave above we don't even know if it was a GET or PUT operation and what was nova trying to do, and we also don't have context, so we cannot tell at which stage of the whole process this happened.
If I remember correctly the high level steps are something like:
- Cinder-volume receives request - Creates new volume - Calls nova to do the volume swap - Nova attaches new volume with os-brick - Nova asks the hypervisor to do the swap - Nova disconnects the original volume - Nova calls cinder telling it that the swap is completed - Cinder detaches the original volume and makes DB changes
Since you are running a 4 years old code it's not easy to see the code differences to see if there was a bug that has been fixed in a later release.
Sorry for not being able to provide actual help. :-(
Cheers, Gorka.
Il giorno lun 7 mar 2022 alle ore 17:44 Ignazio Cassano < ignaziocassano@gmail.com> ha scritto:
Hello, volume.log does not report errors: 2022-03-07 15:41:44.435 159559 INFO cinder.volume.manager [req-68444973-24eb-4b21-95b2-626c9b2e2aa0 5134ed3b93284af5ba2f05d7361edf53 0e760ccde5d24af5a571de40220fbf80 - default default] Created volume successfully. 2022-03-07 15:41:45.358 159099 INFO cinder.volume.manager [req-68444973-24eb-4b21-95b2-626c9b2e2aa0 5134ed3b93284af5ba2f05d7361edf53 0e760ccde5d24af5a571de40220fbf80 - default default] Migrate volume completed successfully. 2022-03-07 15:41:48.970 159559 INFO cinder.volume.manager [req-72ca9242-25e9-4bb0-9bd6-9857f42805ee 5134ed3b93284af5ba2f05d7361edf53 0e760ccde5d24af5a571de40220fbf80 - default default] attachment_update complete
Il giorno lun 7 mar 2022 alle ore 17:25 Eugen Block <eblock@nde.ag> ha scritto:
Hi,
since you're retyping a volume, what do the cinder logs reveal?
Zitat von Ignazio Cassano <ignaziocassano@gmail.com>:
Hello, sometimes the volume retype from a netapp nfs storage to another netapp nfs storage does not work. I do not know the reason but I think something is going wrong in nova:
/var/log/nova/nova-compute.log:2022-02-02 14:23:39.992 194288 ERROR root [req-ebbf749f-359f-4282-94c5-0913cffe97a3 66adb965bef64eaaab2af93ade87e2ca 68feefdf7c5847388534e9214dc13366 - default default] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5655, in _swap_volume\n mountpoint, resize_to)\n', ' File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1647, in swap_volume\n self._disconnect_volume(context, new_connection_info, instance)\n', ' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1644, in swap_volume\n self._swap_volume(guest, disk_dev, conf, resize_to)\n', ' File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1593, in _swap_volume\n raise exception.VolumeRebaseFailed(reason=six.text_type(exc))\n', "VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid: pivot of disk 'vda' requires an active copy job\n"]: VolumeAttachmentNotFound: Volume attachment 2cd820e0-85e8-498d-a62a-800260d0cf31 could not be found
Must I try to change some parameters in nova.conf on compute nodes about volume attachment ?
Thanks Ignazio