[cinder] volume_attachement entries are not getting deleted from DB

Hemant Sonawane hemant.sonawane at itera.io
Tue Dec 13 13:25:14 UTC 2022 

Hello Rajat and Sofia,

Sorry for the late response.

Thanks to both of you.

The logs I shared before are from nova only; we do not see anything apart
from that in nova logs. Here are the full logs for your ready reference.

2022-12-07 10:52:16.189 23213 ERROR nova.volume.cinder
[req-59de985e-9db2-4eb5-a82b-31c64f9758a9 14fd679363724aaf9e9f0503820c0b1e
a95811c12f3449cf8730331fd3be1c69 - default default] Update attachment
failed for attachment cab57e8a-9585-4ca5-9e16-2a75bbf16994. Error: Unable
to update attachment.(Invalid volume: duplicate connectors detected on
volume 7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID:
req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d) Code: 500:
cinderclient.exceptions.ClientException: Unable to update
attachment.(Invalid volume: duplicate connectors detected on volume
7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID:
req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager
[req-59de985e-9db2-4eb5-a82b-31c64f9758a9 14fd679363724aaf9e9f0503820c0b1e
a95811c12f3449cf8730331fd3be1c69 - default default] [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340] Instance failed block device setup:
cinderclient.exceptions.ClientException: Unable to update
attachment.(Invalid volume: duplicate connectors detected on volume
7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID:
req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340] Traceback (most recent call last):

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/nova/compute/manager.py",
line 1976, in _prep_block_device

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]
driver_block_device.attach_block_devices(

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py",
line 874, in attach_block_devices

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     _log_and_attach(device)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py",
line 871, in _log_and_attach

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     bdm.attach(*attach_args,
**attach_kwargs)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py",
line 46, in wrapped

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     ret_val = method(obj, context,
*args, **kwargs)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py",
line 672, in attach

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     self._do_attach(context,
instance, volume, volume_api,

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py",
line 657, in _do_attach

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     self._volume_attach(context,
volume, connector, instance,

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py",
line 571, in _volume_attach

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     connection_info =
volume_api.attachment_update(

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py",
line 396, in wrapper

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     res = method(self, ctx, *args,
**kwargs)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py",
line 447, in wrapper

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     res = method(self, ctx,
attachment_id, *args, **kwargs)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py",
line 875, in attachment_update

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     LOG.error('Update attachment
failed for attachment '

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 227, in __exit__

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     self.force_reraise()

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 200, in force_reraise

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     raise self.value

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py",
line 867, in attachment_update

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     attachment_ref = cinderclient(

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/cinderclient/api_versions.py",
line 423, in substitution

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     return method.func(obj, *args,
**kwargs)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/cinderclient/v3/attachments.py",
line 75, in update

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     resp =
self._update('/attachments/%s' % id, body)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/cinderclient/base.py", line
312, in _update

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     resp, body =
self.api.client.put(url, body=body, **kwargs)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/cinderclient/client.py",
line 220, in put

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     return self._cs_request(url,
'PUT', **kwargs)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/cinderclient/client.py",
line 205, in _cs_request

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     return self.request(url, method,
**kwargs)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]   File
"/var/lib/openstack/lib/python3.8/site-packages/cinderclient/client.py",
line 191, in request

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]     raise
exceptions.from_response(resp, body)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]
cinderclient.exceptions.ClientException: Unable to update
attachment.(Invalid volume: duplicate connectors detected on volume
7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID:
req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d)

2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance:
2f0e87a1-c8a3-48ca-b273-9a2d4206c340]

On Wed, 7 Dec 2022 at 16:48, Rajat Dhasmana <rdhasman at redhat.com> wrote:

> Hi Hemant,
>
> Sorry for getting back to this so late.
> The problem you've described for instance actions like delete, resize or
> shelve not deleting the cinder attachment entries.
> I'm not sure about resize and shelve but deleting an instance should
> delete the volume attachment entry.
> I suggest you check nova logs for the calls it makes to cinder for
> attachment delete operation and see if there are any errors related to it.
> Ideally it should work but due to some issue, there might be failures.
> If you find a failure in nova logs related to attachment calls, you can go
> ahead to check cinder logs (api and volume) to see possible
> issues why the attachment wasn't getting deleted on the cinder side.
>
> -
> Rajat Dhasmana
>
> On Tue, Nov 29, 2022 at 2:07 PM Hemant Sonawane <hemant.sonawane at itera.io>
> wrote:
>
>> Hello Sofia,
>>
>> Thank you for taking it into consideration. Do let me know if you
>> have any questions and updates on the same.
>>
>> On Mon, 28 Nov 2022 at 18:12, Sofia Enriquez <senrique at redhat.com> wrote:
>>
>>> Hi Hemant,
>>>
>>> Thanks for reporting this issue on the bug tracker
>>> https://bugs.launchpad.net/cinder/+bug/1998083
>>>
>>> I did a quick search and no problems with shelving operations have been
>>> reported for at least the last two years.I'll bring this bug to the cinder
>>> bug meeting this week.
>>>
>>> Thanks
>>> Sofia
>>>
>>> On Fri, Nov 25, 2022 at 1:15 PM Hemant Sonawane <
>>> hemant.sonawane at itera.io> wrote:
>>>
>>>> Hi Rajat,
>>>> It's not about deleting attachments entries but the normal operations
>>>> from horizon or via cli does not work because of that. So it really needs
>>>> to be fixed to perform resize, shelve unshelve operations.
>>>>
>>>> Here are the detailed attachment entries you can see for the shelved
>>>> instance.
>>>>
>>>>
>>>>
>>>> +--------------------------------------+--------------------------------------+--------------------------+----------------------------------
>>>> *----+---------------+-----------------------------------------**│·*
>>>>
>>>> *--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
>>>>             │·*
>>>>
>>>> *| id                                   | volume_id
>>>>         | attached_host            | instance_uuid                        |
>>>> attach_status | connector                               │·*
>>>>
>>>> *
>>>>
>>>>                                           |             │·*
>>>>
>>>>
>>>> *+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·*
>>>>
>>>> *--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
>>>>             │·*
>>>>
>>>> *| 8daddacc-8fc8-4d2b-a738-d05deb20049f |
>>>> 67ea3a39-78b8-4d04-a280-166acdc90b8a | nfv1compute43.nfv1.o2.cz
>>>> <http://nfv1compute43.nfv1.o2.cz> | 9266a2d7-9721-4994-a6b5-6b3290862dc6 |
>>>> attached      | {"platform": "x86_64", "os_type": "linux│·*
>>>>
>>>> *", "ip": "10.42.168.87", "host": "nfv1compute43.nfv1.o2.cz
>>>> <http://nfv1compute43.nfv1.o2.cz>", "multipath": false, "do_local_attach":
>>>> false, "system uuid": "65917e4f-c8c4-a2af-ec11-fe353e13f4dd", "mountpoint":
>>>> "/dev/vda"} |             │·*
>>>>
>>>> *| d3278543-4920-42b7-b217-0858e986fcce |
>>>> 67ea3a39-78b8-4d04-a280-166acdc90b8a | NULL                     |
>>>> 9266a2d7-9721-4994-a6b5-6b3290862dc6 | reserved      | NULL
>>>>                     │·*
>>>>
>>>> *
>>>>
>>>>                                           |             │·*
>>>>
>>>>
>>>> *+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·*
>>>>
>>>> *--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
>>>>             │·*
>>>>
>>>> *2 rows in set (0.00 sec) *
>>>>
>>>>
>>>> for e.g if I would like to unshelve this instance it wont work as it
>>>> has a duplicate entry in cinder db for the attachment. So i have to delete
>>>> it manually from db or via cli
>>>>
>>>> *root at master01:/home/hemant# cinder --os-volume-api-version 3.27
>>>> attachment-list --all | grep 67ea3a39-78b8-4d04-a280-166acdc90b8a
>>>>                                                               │·*
>>>>
>>>> *| 8daddacc-8fc8-4d2b-a738-d05deb20049f |
>>>> 67ea3a39-78b8-4d04-a280-166acdc90b8a | attached |
>>>> 9266a2d7-9721-4994-a6b5-6b3290862dc6 |
>>>>                                     │·*
>>>>
>>>> *| d3278543-4920-42b7-b217-0858e986fcce |
>>>> 67ea3a39-78b8-4d04-a280-166acdc90b8a** | reserved |
>>>> 9266a2d7-9721-4994-a6b5-6b3290862dc6 |*
>>>>
>>>> *cinder --os-volume-api-version 3.27
>>>> attachment-delete 8daddacc-8fc8-4d2b-a738-d05deb20049f*
>>>>
>>>> this is the only choice I have if I would like to unshelve vm. But this
>>>> is not a good approach for production envs. I hope you understand me.
>>>> Please feel free to ask me anything if you don't understand.
>>>>
>>>>
>>>>
>>>> On Fri, 25 Nov 2022 at 13:20, Rajat Dhasmana <rdhasman at redhat.com>
>>>> wrote:
>>>>
>>>>> Hi Hemant,
>>>>>
>>>>> If your final goal is to delete the attachment entries in the cinder
>>>>> DB, we have attachment APIs to perform these tasks. The command useful for
>>>>> you is attachment list[1] and attachment delete[2].
>>>>> Make sure you pass the right microversion i.e. 3.27 to be able to
>>>>> execute these operations.
>>>>>
>>>>> Eg:
>>>>> cinder --os-volume-api-version 3.27 attachment-list
>>>>>
>>>>> [1]
>>>>> https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinder-attachment-list
>>>>> [2]
>>>>> https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinder-attachment-delete
>>>>>
>>>>> On Fri, Nov 25, 2022 at 5:44 PM Hemant Sonawane <
>>>>> hemant.sonawane at itera.io> wrote:
>>>>>
>>>>>> Hello
>>>>>> I am using wallaby release openstack and having  issues with cinder
>>>>>> volumes as once I try to delete, resize or unshelve the shelved vms the
>>>>>> volume_attachement entries do not get deleted in cinder db and therefore
>>>>>> the above mentioned operations fail every time. I have to delete these
>>>>>> volume_attachement entries manually then it works. Is there any way to fix
>>>>>> this issue ?
>>>>>>
>>>>>> nova-compute logs:
>>>>>>
>>>>>> cinderclient.exceptions.ClientException: Unable to update
>>>>>> attachment.(Invalid volume: duplicate connectors detected on volume
>>>>>>
>>>>>> Help will be really appreciated Thanks !
>>>>>> --
>>>>>> Thanks and Regards,
>>>>>>
>>>>>> Hemant Sonawane
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Thanks and Regards,
>>>>
>>>> Hemant Sonawane
>>>>
>>>>
>>>
>>> --
>>>
>>> Sofía Enriquez
>>>
>>> she/her
>>>
>>> Software Engineer
>>>
>>> Red Hat PnT <https://www.redhat.com>
>>>
>>> IRC: @enriquetaso
>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>> <https://www.facebook.com/RedHatInc>
>>> <https://www.redhat.com>
>>>
>>>
>>
>> --
>> Thanks and Regards,
>>
>> Hemant Sonawane
>>
>>

-- 
Thanks and Regards,

Hemant Sonawane
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20221213/c96da449/attachment-0001.htm>

More information about the openstack-discuss mailing list