Open Stack

On 07/03, Ignazio Cassano wrote:
> Hello again I found the error but it is in cinder api log:
> 2022-03-07 15:42:57.330 156916 INFO cinder.api.openstack.wsgi
> [req-01eb2613-4aad-4809-ac91-037de3148fbb 5134ed3b93284af5ba2f05d7361edf53
> 0e760ccde5d24af5a571de40220fbf80 - default default]
> http://10.138.144.16:8776/v3/0e760ccde5d24af5a571de40220fbf80/attachments/04ae01f6-e164-40cb-8d31-63f78e69c8db
> returned with HTTP 404
>
> and in nova compute log:
> 15:42:57.502 280431 ERROR oslo_messaging.rpc.server
> [req-dfeac1f0-870b-4aef-af40-cda52f4552e0 5134ed3b93284af5ba2f05d7361edf53
> 0e760ccde5d24af5a571de40220fbf80 - default default] Exception during
> message handling: VolumeAttachmentNotFound: Volume attachment
> 04ae01f6-e164-40cb-8d31-63f78e69c8db could not be found.
>

Hi,

Retyping with migration of attached volumes is a complex flow, so doing
the RCA usually requires full logs of ALL involved services:

- All nova-api, cinder-api, cinder-scheduler services
- nova-cpu of the host with the instance
- Cinder-volume service

For example in the logs you gave above we don't even know if it was a
GET or PUT operation and what was nova trying to do, and we also don't
have context, so we cannot tell at which stage of the whole process this
happened.

If I remember correctly the high level steps are something like:

- Cinder-volume receives request
- Creates new volume
- Calls nova to do the volume swap
- Nova attaches new volume with os-brick
- Nova asks the hypervisor to do the swap
- Nova disconnects the original volume
- Nova calls cinder telling it that the swap is completed
- Cinder detaches the original volume and makes DB changes

Since you are running a 4 years old code it's not easy to see the code
differences to see if there was a bug that has been fixed in a later
release.

Sorry for not being able to provide actual help.  :-(

Cheers,
Gorka.


> Il giorno lun 7 mar 2022 alle ore 17:44 Ignazio Cassano <
> ignaziocassano at gmail.com> ha scritto:
>
> > Hello, volume.log does not report errors:
> > 2022-03-07 15:41:44.435 159559 INFO cinder.volume.manager
> > [req-68444973-24eb-4b21-95b2-626c9b2e2aa0 5134ed3b93284af5ba2f05d7361edf53
> > 0e760ccde5d24af5a571de40220fbf80 - default default] Created volume
> > successfully.
> > 2022-03-07 15:41:45.358 159099 INFO cinder.volume.manager
> > [req-68444973-24eb-4b21-95b2-626c9b2e2aa0 5134ed3b93284af5ba2f05d7361edf53
> > 0e760ccde5d24af5a571de40220fbf80 - default default] Migrate volume
> > completed successfully.
> > 2022-03-07 15:41:48.970 159559 INFO cinder.volume.manager
> > [req-72ca9242-25e9-4bb0-9bd6-9857f42805ee 5134ed3b93284af5ba2f05d7361edf53
> > 0e760ccde5d24af5a571de40220fbf80 - default default] attachment_update
> > complete
> >
> > Il giorno lun 7 mar 2022 alle ore 17:25 Eugen Block <eblock at nde.ag> ha
> > scritto:
> >
> >> Hi,
> >>
> >> since you're retyping a volume, what do the cinder logs reveal?
> >>
> >>
> >> Zitat von Ignazio Cassano <ignaziocassano at gmail.com>:
> >>
> >> > Hello, sometimes the volume retype from a netapp nfs storage to another
> >> > netapp nfs storage does not work.
> >> > I do not know the reason but I think something is going wrong in nova:
> >> >
> >> > /var/log/nova/nova-compute.log:2022-02-02 14:23:39.992 194288 ERROR root
> >> > [req-ebbf749f-359f-4282-94c5-0913cffe97a3
> >> 66adb965bef64eaaab2af93ade87e2ca
> >> > 68feefdf7c5847388534e9214dc13366 - default default] Original exception
> >> > being dropped: ['Traceback (most recent call last):\n', '  File
> >> > "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5655,
> >> in
> >> > _swap_volume\n    mountpoint, resize_to)\n', '  File
> >> > "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
> >> 1647,
> >> > in swap_volume\n    self._disconnect_volume(context,
> >> new_connection_info,
> >> > instance)\n', '  File
> >> > "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
> >> > __exit__\n    self.force_reraise()\n', '  File
> >> > "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
> >> > force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n', '
> >> File
> >> > "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
> >> 1644,
> >> > in swap_volume\n    self._swap_volume(guest, disk_dev, conf,
> >> resize_to)\n',
> >> > '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
> >> > line 1593, in _swap_volume\n    raise
> >> > exception.VolumeRebaseFailed(reason=six.text_type(exc))\n',
> >> > "VolumeRebaseFailed: Volume rebase failed: Requested operation is not
> >> > valid: pivot of disk 'vda' requires an active copy job\n"]:
> >> > VolumeAttachmentNotFound: Volume attachment
> >> > 2cd820e0-85e8-498d-a62a-800260d0cf31 could not be found
> >> >
> >> >
> >> > Must I try to change some parameters in nova.conf on compute nodes about
> >> > volume attachment ?
> >> >
> >> > Thanks
> >> > Ignazio
> >>
> >>
> >>
> >>
> >>