Hello Rajat and Sofia,

Sorry for the late response.

Thanks to both of you. 

The logs I shared before are from nova only; we do not see anything apart from that in nova logs. Here are the full logs for your ready reference.

2022-12-07 10:52:16.189 23213 ERROR nova.volume.cinder [req-59de985e-9db2-4eb5-a82b-31c64f9758a9 14fd679363724aaf9e9f0503820c0b1e a95811c12f3449cf8730331fd3be1c69 - default default] Update attachment failed for attachment cab57e8a-9585-4ca5-9e16-2a75bbf16994. Error: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID: req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d) Code: 500: cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID: req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [req-59de985e-9db2-4eb5-a82b-31c64f9758a9 14fd679363724aaf9e9f0503820c0b1e a95811c12f3449cf8730331fd3be1c69 - default default] [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] Instance failed block device setup: cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID: req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] Traceback (most recent call last):

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/compute/manager.py", line 1976, in _prep_block_device

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     driver_block_device.attach_block_devices(

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 874, in attach_block_devices

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     _log_and_attach(device)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 871, in _log_and_attach

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     bdm.attach(*attach_args, **attach_kwargs)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 46, in wrapped

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     ret_val = method(obj, context, *args, **kwargs)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 672, in attach

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     self._do_attach(context, instance, volume, volume_api,

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 657, in _do_attach

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     self._volume_attach(context, volume, connector, instance,

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 571, in _volume_attach

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     connection_info = volume_api.attachment_update(

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 396, in wrapper

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     res = method(self, ctx, *args, **kwargs)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 447, in wrapper

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     res = method(self, ctx, attachment_id, *args, **kwargs)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 875, in attachment_update

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     LOG.error('Update attachment failed for attachment '

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     self.force_reraise()

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     raise self.value

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 867, in attachment_update

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     attachment_ref = cinderclient(

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/api_versions.py", line 423, in substitution

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     return method.func(obj, *args, **kwargs)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/v3/attachments.py", line 75, in update

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     resp = self._update('/attachments/%s' % id, body)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/base.py", line 312, in _update

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     resp, body = self.api.client.put(url, body=body, **kwargs)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/client.py", line 220, in put

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     return self._cs_request(url, 'PUT', **kwargs)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/client.py", line 205, in _cs_request

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     return self.request(url, method, **kwargs)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/client.py", line 191, in request

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     raise exceptions.from_response(resp, body)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID: req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] 


On Wed, 7 Dec 2022 at 16:48, Rajat Dhasmana <rdhasman@redhat.com> wrote:
Hi Hemant,

Sorry for getting back to this so late.
The problem you've described for instance actions like delete, resize or shelve not deleting the cinder attachment entries.
I'm not sure about resize and shelve but deleting an instance should delete the volume attachment entry.
I suggest you check nova logs for the calls it makes to cinder for attachment delete operation and see if there are any errors related to it.
Ideally it should work but due to some issue, there might be failures.
If you find a failure in nova logs related to attachment calls, you can go ahead to check cinder logs (api and volume) to see possible
issues why the attachment wasn't getting deleted on the cinder side.

-
Rajat Dhasmana

On Tue, Nov 29, 2022 at 2:07 PM Hemant Sonawane <hemant.sonawane@itera.io> wrote:
Hello Sofia, 

Thank you for taking it into consideration. Do let me know if you have any questions and updates on the same. 

On Mon, 28 Nov 2022 at 18:12, Sofia Enriquez <senrique@redhat.com> wrote:
Hi Hemant,

Thanks for reporting this issue on the bug tracker https://bugs.launchpad.net/cinder/+bug/1998083

I did a quick search and no problems with shelving operations have been reported for at least the last two years.I'll bring this bug to the cinder bug meeting this week.

Thanks
Sofia

On Fri, Nov 25, 2022 at 1:15 PM Hemant Sonawane <hemant.sonawane@itera.io> wrote:
Hi Rajat,
It's not about deleting attachments entries but the normal operations from horizon or via cli does not work because of that. So it really needs to be fixed to perform resize, shelve unshelve operations. 

Here are the detailed attachment entries you can see for the shelved instance.


+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------·

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+             ·

| id                                   | volume_id                            | attached_host            | instance_uuid                        | attach_status | connector                               ·

                                                                                                                                                                                            |             ·

+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------·

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+             ·

| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | nfv1compute43.nfv1.o2.cz | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | attached      | {"platform": "x86_64", "os_type": "linux·

", "ip": "10.42.168.87", "host": "nfv1compute43.nfv1.o2.cz", "multipath": false, "do_local_attach": false, "system uuid": "65917e4f-c8c4-a2af-ec11-fe353e13f4dd", "mountpoint": "/dev/vda"} |             ·

| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a | NULL                     | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | reserved      | NULL                                    ·

                                                                                                                                                                                            |             ·

+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------·

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+             ·

2 rows in set (0.00 sec)    


for e.g if I would like to unshelve this instance it wont work as it has a duplicate entry in cinder db for the attachment. So i have to delete it manually from db or via cli 

root@master01:/home/hemant# cinder --os-volume-api-version 3.27 attachment-list --all | grep 67ea3a39-78b8-4d04-a280-166acdc90b8a                                                                         ·

| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | attached | 9266a2d7-9721-4994-a6b5-6b3290862dc6 |                                                                         ·

| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a | reserved | 9266a2d7-9721-4994-a6b5-6b3290862dc6 |

cinder --os-volume-api-version 3.27 attachment-delete 8daddacc-8fc8-4d2b-a738-d05deb20049f

this is the only choice I have if I would like to unshelve vm. But this is not a good approach for production envs. I hope you understand me. Please feel free to ask me anything if you don't understand. 



On Fri, 25 Nov 2022 at 13:20, Rajat Dhasmana <rdhasman@redhat.com> wrote:
Hi Hemant,

If your final goal is to delete the attachment entries in the cinder DB, we have attachment APIs to perform these tasks. The command useful for you is attachment list[1] and attachment delete[2].
Make sure you pass the right microversion i.e. 3.27 to be able to execute these operations.

Eg:
cinder --os-volume-api-version 3.27 attachment-list


On Fri, Nov 25, 2022 at 5:44 PM Hemant Sonawane <hemant.sonawane@itera.io> wrote:
Hello  
I am using wallaby release openstack and having  issues with cinder volumes as once I try to delete, resize or unshelve the shelved vms the volume_attachement entries do not get deleted in cinder db and therefore the above mentioned operations fail every time. I have to delete these volume_attachement entries manually then it works. Is there any way to fix this issue ?

nova-compute logs:

cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume

Help will be really appreciated Thanks !
--
Thanks and Regards,

Hemant Sonawane



--
Thanks and Regards,

Hemant Sonawane



--

Sofía Enriquez

she/her

Software Engineer

Red Hat PnT

IRC: @enriquetaso




--
Thanks and Regards,

Hemant Sonawane



--
Thanks and Regards,

Hemant Sonawane