2022-12-07 10:52:16.189 23213 ERROR nova.volume.cinder [req-59de985e-9db2-4eb5-a82b-31c64f9758a9 14fd679363724aaf9e9f0503820c0b1e a95811c12f3449cf8730331fd3be1c69 - default default] Update attachment failed for attachment cab57e8a-9585-4ca5-9e16-2a75bbf16994. Error: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID: req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d) Code: 500: cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID: req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d)
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [req-59de985e-9db2-4eb5-a82b-31c64f9758a9 14fd679363724aaf9e9f0503820c0b1e a95811c12f3449cf8730331fd3be1c69 - default default] [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] Instance failed block device setup: cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID: req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d)
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] Traceback (most recent call last):
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/compute/manager.py", line 1976, in _prep_block_device
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] driver_block_device.attach_block_devices(
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 874, in attach_block_devices
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] _log_and_attach(device)
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 871, in _log_and_attach
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] bdm.attach(*attach_args, **attach_kwargs)
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 46, in wrapped
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] ret_val = method(obj, context, *args, **kwargs)
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 672, in attach
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] self._do_attach(context, instance, volume, volume_api,
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 657, in _do_attach
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] self._volume_attach(context, volume, connector, instance,
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 571, in _volume_attach
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] connection_info = volume_api.attachment_update(
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 396, in wrapper
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] res = method(self, ctx, *args, **kwargs)
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 447, in wrapper
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] res = method(self, ctx, attachment_id, *args, **kwargs)
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 875, in attachment_update
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] LOG.error('Update attachment failed for attachment '
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] self.force_reraise()
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] raise self.value
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 867, in attachment_update
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] attachment_ref = cinderclient(
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/api_versions.py", line 423, in substitution
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] return method.func(obj, *args, **kwargs)
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/v3/attachments.py", line 75, in update
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] resp = self._update('/attachments/%s' % id, body)
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/base.py", line 312, in _update
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] resp, body = self.api.client.put(url, body=body, **kwargs)
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/client.py", line 220, in put
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] return self._cs_request(url, 'PUT', **kwargs)
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/client.py", line 205, in _cs_request
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] return self.request(url, method, **kwargs)
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/client.py", line 191, in request
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] raise exceptions.from_response(resp, body)
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID: req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d)
2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]
Hi Hemant,Sorry for getting back to this so late.The problem you've described for instance actions like delete, resize or shelve not deleting the cinder attachment entries.I'm not sure about resize and shelve but deleting an instance should delete the volume attachment entry.I suggest you check nova logs for the calls it makes to cinder for attachment delete operation and see if there are any errors related to it.Ideally it should work but due to some issue, there might be failures.If you find a failure in nova logs related to attachment calls, you can go ahead to check cinder logs (api and volume) to see possibleissues why the attachment wasn't getting deleted on the cinder side.-Rajat DhasmanaOn Tue, Nov 29, 2022 at 2:07 PM Hemant Sonawane <hemant.sonawane@itera.io> wrote:Hello Sofia,
Thank you for taking it into consideration. Do let me know if you have any questions and updates on the same.On Mon, 28 Nov 2022 at 18:12, Sofia Enriquez <senrique@redhat.com> wrote:Hi Hemant,Thanks for reporting this issue on the bug tracker https://bugs.launchpad.net/cinder/+bug/1998083I did a quick search and no problems with shelving operations have been reported for at least the last two years.I'll bring this bug to the cinder bug meeting this week.ThanksSofiaOn Fri, Nov 25, 2022 at 1:15 PM Hemant Sonawane <hemant.sonawane@itera.io> wrote:Hi Rajat,
It's not about deleting attachments entries but the normal operations from horizon or via cli does not work because of that. So it really needs to be fixed to perform resize, shelve unshelve operations.
Here are the detailed attachment entries you can see for the shelved instance.
+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·
| id | volume_id | attached_host | instance_uuid | attach_status | connector │·
| │·
+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·
| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | nfv1compute43.nfv1.o2.cz | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | attached | {"platform": "x86_64", "os_type": "linux│·
", "ip": "10.42.168.87", "host": "nfv1compute43.nfv1.o2.cz", "multipath": false, "do_local_attach": false, "system uuid": "65917e4f-c8c4-a2af-ec11-fe353e13f4dd", "mountpoint": "/dev/vda"} | │·
| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a | NULL | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | reserved | NULL │·
| │·
+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·
2 rows in set (0.00 sec)
for e.g if I would like to unshelve this instance it wont work as it has a duplicate entry in cinder db for the attachment. So i have to delete it manually from db or via cli
root@master01:/home/hemant# cinder --os-volume-api-version 3.27 attachment-list --all | grep 67ea3a39-78b8-4d04-a280-166acdc90b8a │·
| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | attached | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | │·
| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a | reserved | 9266a2d7-9721-4994-a6b5-6b3290862dc6 |
cinder --os-volume-api-version 3.27 attachment-delete 8daddacc-8fc8-4d2b-a738-d05deb20049f
this is the only choice I have if I would like to unshelve vm. But this is not a good approach for production envs. I hope you understand me. Please feel free to ask me anything if you don't understand.On Fri, 25 Nov 2022 at 13:20, Rajat Dhasmana <rdhasman@redhat.com> wrote:Hi Hemant,If your final goal is to delete the attachment entries in the cinder DB, we have attachment APIs to perform these tasks. The command useful for you is attachment list[1] and attachment delete[2].Make sure you pass the right microversion i.e. 3.27 to be able to execute these operations.Eg:cinder --os-volume-api-version 3.27 attachment-listOn Fri, Nov 25, 2022 at 5:44 PM Hemant Sonawane <hemant.sonawane@itera.io> wrote:Hello
I am using wallaby release openstack and having issues with cinder volumes as once I try to delete, resize or unshelve the shelved vms the volume_attachement entries do not get deleted in cinder db and therefore the above mentioned operations fail every time. I have to delete these volume_attachement entries manually then it works. Is there any way to fix this issue ?nova-compute logs:--
cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume
Help will be really appreciated Thanks !Thanks and Regards,Hemant Sonawane--Thanks and Regards,Hemant Sonawane
--
--Thanks and Regards,Hemant Sonawane