[cinder] volume_attachement entries are not getting deleted from DB
Hello I am using wallaby release openstack and having issues with cinder volumes as once I try to delete, resize or unshelve the shelved vms the volume_attachement entries do not get deleted in cinder db and therefore the above mentioned operations fail every time. I have to delete these volume_attachement entries manually then it works. Is there any way to fix this issue ? nova-compute logs: cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume Help will be really appreciated Thanks ! -- Thanks and Regards, Hemant Sonawane
Hi Hemant, If your final goal is to delete the attachment entries in the cinder DB, we have attachment APIs to perform these tasks. The command useful for you is attachment list[1] and attachment delete[2]. Make sure you pass the right microversion i.e. 3.27 to be able to execute these operations. Eg: cinder --os-volume-api-version 3.27 attachment-list [1] https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinde... [2] https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinde... On Fri, Nov 25, 2022 at 5:44 PM Hemant Sonawane <hemant.sonawane@itera.io> wrote:
Hello I am using wallaby release openstack and having issues with cinder volumes as once I try to delete, resize or unshelve the shelved vms the volume_attachement entries do not get deleted in cinder db and therefore the above mentioned operations fail every time. I have to delete these volume_attachement entries manually then it works. Is there any way to fix this issue ?
nova-compute logs:
cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume
Help will be really appreciated Thanks ! -- Thanks and Regards,
Hemant Sonawane
Hi Rajat, It's not about deleting attachments entries but the normal operations from horizon or via cli does not work because of that. So it really needs to be fixed to perform resize, shelve unshelve operations. Here are the detailed attachment entries you can see for the shelved instance. +--------------------------------------+--------------------------------------+--------------------------+---------------------------------- *----+---------------+-----------------------------------------**│·* *--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·* *| id | volume_id | attached_host | instance_uuid | attach_status | connector │·* * | │·* *+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·* *--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·* *| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | nfv1compute43.nfv1.o2.cz <http://nfv1compute43.nfv1.o2.cz> | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | attached | {"platform": "x86_64", "os_type": "linux│·* *", "ip": "10.42.168.87", "host": "nfv1compute43.nfv1.o2.cz <http://nfv1compute43.nfv1.o2.cz>", "multipath": false, "do_local_attach": false, "system uuid": "65917e4f-c8c4-a2af-ec11-fe353e13f4dd", "mountpoint": "/dev/vda"} | │·* *| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a | NULL | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | reserved | NULL │·* * | │·* *+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·* *--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·* *2 rows in set (0.00 sec) * for e.g if I would like to unshelve this instance it wont work as it has a duplicate entry in cinder db for the attachment. So i have to delete it manually from db or via cli *root@master01:/home/hemant# cinder --os-volume-api-version 3.27 attachment-list --all | grep 67ea3a39-78b8-4d04-a280-166acdc90b8a │·* *| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | attached | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | │·* *| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a** | reserved | 9266a2d7-9721-4994-a6b5-6b3290862dc6 |* *cinder --os-volume-api-version 3.27 attachment-delete 8daddacc-8fc8-4d2b-a738-d05deb20049f* this is the only choice I have if I would like to unshelve vm. But this is not a good approach for production envs. I hope you understand me. Please feel free to ask me anything if you don't understand. On Fri, 25 Nov 2022 at 13:20, Rajat Dhasmana <rdhasman@redhat.com> wrote:
Hi Hemant,
If your final goal is to delete the attachment entries in the cinder DB, we have attachment APIs to perform these tasks. The command useful for you is attachment list[1] and attachment delete[2]. Make sure you pass the right microversion i.e. 3.27 to be able to execute these operations.
Eg: cinder --os-volume-api-version 3.27 attachment-list
[1] https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinde... [2] https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinde...
On Fri, Nov 25, 2022 at 5:44 PM Hemant Sonawane <hemant.sonawane@itera.io> wrote:
Hello I am using wallaby release openstack and having issues with cinder volumes as once I try to delete, resize or unshelve the shelved vms the volume_attachement entries do not get deleted in cinder db and therefore the above mentioned operations fail every time. I have to delete these volume_attachement entries manually then it works. Is there any way to fix this issue ?
nova-compute logs:
cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume
Help will be really appreciated Thanks ! -- Thanks and Regards,
Hemant Sonawane
-- Thanks and Regards, Hemant Sonawane
Hi Hemant, Thanks for reporting this issue on the bug tracker https://bugs.launchpad.net/cinder/+bug/1998083 I did a quick search and no problems with shelving operations have been reported for at least the last two years.I'll bring this bug to the cinder bug meeting this week. Thanks Sofia On Fri, Nov 25, 2022 at 1:15 PM Hemant Sonawane <hemant.sonawane@itera.io> wrote:
Hi Rajat, It's not about deleting attachments entries but the normal operations from horizon or via cli does not work because of that. So it really needs to be fixed to perform resize, shelve unshelve operations.
Here are the detailed attachment entries you can see for the shelved instance.
+--------------------------------------+--------------------------------------+--------------------------+---------------------------------- *----+---------------+-----------------------------------------**│·*
*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·*
*| id | volume_id | attached_host | instance_uuid | attach_status | connector │·*
*
| │·*
*+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·*
*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·*
*| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | nfv1compute43.nfv1.o2.cz <http://nfv1compute43.nfv1.o2.cz> | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | attached | {"platform": "x86_64", "os_type": "linux│·*
*", "ip": "10.42.168.87", "host": "nfv1compute43.nfv1.o2.cz <http://nfv1compute43.nfv1.o2.cz>", "multipath": false, "do_local_attach": false, "system uuid": "65917e4f-c8c4-a2af-ec11-fe353e13f4dd", "mountpoint": "/dev/vda"} | │·*
*| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a | NULL | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | reserved | NULL │·*
*
| │·*
*+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·*
*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·*
*2 rows in set (0.00 sec) *
for e.g if I would like to unshelve this instance it wont work as it has a duplicate entry in cinder db for the attachment. So i have to delete it manually from db or via cli
*root@master01:/home/hemant# cinder --os-volume-api-version 3.27 attachment-list --all | grep 67ea3a39-78b8-4d04-a280-166acdc90b8a │·*
*| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | attached | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | │·*
*| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a** | reserved | 9266a2d7-9721-4994-a6b5-6b3290862dc6 |*
*cinder --os-volume-api-version 3.27 attachment-delete 8daddacc-8fc8-4d2b-a738-d05deb20049f*
this is the only choice I have if I would like to unshelve vm. But this is not a good approach for production envs. I hope you understand me. Please feel free to ask me anything if you don't understand.
On Fri, 25 Nov 2022 at 13:20, Rajat Dhasmana <rdhasman@redhat.com> wrote:
Hi Hemant,
If your final goal is to delete the attachment entries in the cinder DB, we have attachment APIs to perform these tasks. The command useful for you is attachment list[1] and attachment delete[2]. Make sure you pass the right microversion i.e. 3.27 to be able to execute these operations.
Eg: cinder --os-volume-api-version 3.27 attachment-list
[1] https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinde... [2] https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinde...
On Fri, Nov 25, 2022 at 5:44 PM Hemant Sonawane <hemant.sonawane@itera.io> wrote:
Hello I am using wallaby release openstack and having issues with cinder volumes as once I try to delete, resize or unshelve the shelved vms the volume_attachement entries do not get deleted in cinder db and therefore the above mentioned operations fail every time. I have to delete these volume_attachement entries manually then it works. Is there any way to fix this issue ?
nova-compute logs:
cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume
Help will be really appreciated Thanks ! -- Thanks and Regards,
Hemant Sonawane
-- Thanks and Regards,
Hemant Sonawane
-- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>
Hello Sofia, Thank you for taking it into consideration. Do let me know if you have any questions and updates on the same. On Mon, 28 Nov 2022 at 18:12, Sofia Enriquez <senrique@redhat.com> wrote:
Hi Hemant,
Thanks for reporting this issue on the bug tracker https://bugs.launchpad.net/cinder/+bug/1998083
I did a quick search and no problems with shelving operations have been reported for at least the last two years.I'll bring this bug to the cinder bug meeting this week.
Thanks Sofia
On Fri, Nov 25, 2022 at 1:15 PM Hemant Sonawane <hemant.sonawane@itera.io> wrote:
Hi Rajat, It's not about deleting attachments entries but the normal operations from horizon or via cli does not work because of that. So it really needs to be fixed to perform resize, shelve unshelve operations.
Here are the detailed attachment entries you can see for the shelved instance.
+--------------------------------------+--------------------------------------+--------------------------+---------------------------------- *----+---------------+-----------------------------------------**│·*
*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·*
*| id | volume_id | attached_host | instance_uuid | attach_status | connector │·*
*
| │·*
*+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·*
*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·*
*| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | nfv1compute43.nfv1.o2.cz <http://nfv1compute43.nfv1.o2.cz> | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | attached | {"platform": "x86_64", "os_type": "linux│·*
*", "ip": "10.42.168.87", "host": "nfv1compute43.nfv1.o2.cz <http://nfv1compute43.nfv1.o2.cz>", "multipath": false, "do_local_attach": false, "system uuid": "65917e4f-c8c4-a2af-ec11-fe353e13f4dd", "mountpoint": "/dev/vda"} | │·*
*| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a | NULL | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | reserved | NULL │·*
*
| │·*
*+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·*
*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·*
*2 rows in set (0.00 sec) *
for e.g if I would like to unshelve this instance it wont work as it has a duplicate entry in cinder db for the attachment. So i have to delete it manually from db or via cli
*root@master01:/home/hemant# cinder --os-volume-api-version 3.27 attachment-list --all | grep 67ea3a39-78b8-4d04-a280-166acdc90b8a │·*
*| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | attached | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | │·*
*| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a** | reserved | 9266a2d7-9721-4994-a6b5-6b3290862dc6 |*
*cinder --os-volume-api-version 3.27 attachment-delete 8daddacc-8fc8-4d2b-a738-d05deb20049f*
this is the only choice I have if I would like to unshelve vm. But this is not a good approach for production envs. I hope you understand me. Please feel free to ask me anything if you don't understand.
On Fri, 25 Nov 2022 at 13:20, Rajat Dhasmana <rdhasman@redhat.com> wrote:
Hi Hemant,
If your final goal is to delete the attachment entries in the cinder DB, we have attachment APIs to perform these tasks. The command useful for you is attachment list[1] and attachment delete[2]. Make sure you pass the right microversion i.e. 3.27 to be able to execute these operations.
Eg: cinder --os-volume-api-version 3.27 attachment-list
[1] https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinde... [2] https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinde...
On Fri, Nov 25, 2022 at 5:44 PM Hemant Sonawane < hemant.sonawane@itera.io> wrote:
Hello I am using wallaby release openstack and having issues with cinder volumes as once I try to delete, resize or unshelve the shelved vms the volume_attachement entries do not get deleted in cinder db and therefore the above mentioned operations fail every time. I have to delete these volume_attachement entries manually then it works. Is there any way to fix this issue ?
nova-compute logs:
cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume
Help will be really appreciated Thanks ! -- Thanks and Regards,
Hemant Sonawane
-- Thanks and Regards,
Hemant Sonawane
--
Sofía Enriquez
she/her
Software Engineer
Red Hat PnT <https://www.redhat.com>
IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>
-- Thanks and Regards, Hemant Sonawane
Hi Hemant, Sorry for getting back to this so late. The problem you've described for instance actions like delete, resize or shelve not deleting the cinder attachment entries. I'm not sure about resize and shelve but deleting an instance should delete the volume attachment entry. I suggest you check nova logs for the calls it makes to cinder for attachment delete operation and see if there are any errors related to it. Ideally it should work but due to some issue, there might be failures. If you find a failure in nova logs related to attachment calls, you can go ahead to check cinder logs (api and volume) to see possible issues why the attachment wasn't getting deleted on the cinder side. - Rajat Dhasmana On Tue, Nov 29, 2022 at 2:07 PM Hemant Sonawane <hemant.sonawane@itera.io> wrote:
Hello Sofia,
Thank you for taking it into consideration. Do let me know if you have any questions and updates on the same.
On Mon, 28 Nov 2022 at 18:12, Sofia Enriquez <senrique@redhat.com> wrote:
Hi Hemant,
Thanks for reporting this issue on the bug tracker https://bugs.launchpad.net/cinder/+bug/1998083
I did a quick search and no problems with shelving operations have been reported for at least the last two years.I'll bring this bug to the cinder bug meeting this week.
Thanks Sofia
On Fri, Nov 25, 2022 at 1:15 PM Hemant Sonawane <hemant.sonawane@itera.io> wrote:
Hi Rajat, It's not about deleting attachments entries but the normal operations from horizon or via cli does not work because of that. So it really needs to be fixed to perform resize, shelve unshelve operations.
Here are the detailed attachment entries you can see for the shelved instance.
+--------------------------------------+--------------------------------------+--------------------------+---------------------------------- *----+---------------+-----------------------------------------**│·*
*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·*
*| id | volume_id | attached_host | instance_uuid | attach_status | connector │·*
*
| │·*
*+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·*
*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·*
*| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | nfv1compute43.nfv1.o2.cz <http://nfv1compute43.nfv1.o2.cz> | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | attached | {"platform": "x86_64", "os_type": "linux│·*
*", "ip": "10.42.168.87", "host": "nfv1compute43.nfv1.o2.cz <http://nfv1compute43.nfv1.o2.cz>", "multipath": false, "do_local_attach": false, "system uuid": "65917e4f-c8c4-a2af-ec11-fe353e13f4dd", "mountpoint": "/dev/vda"} | │·*
*| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a | NULL | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | reserved | NULL │·*
*
| │·*
*+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·*
*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·*
*2 rows in set (0.00 sec) *
for e.g if I would like to unshelve this instance it wont work as it has a duplicate entry in cinder db for the attachment. So i have to delete it manually from db or via cli
*root@master01:/home/hemant# cinder --os-volume-api-version 3.27 attachment-list --all | grep 67ea3a39-78b8-4d04-a280-166acdc90b8a │·*
*| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | attached | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | │·*
*| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a** | reserved | 9266a2d7-9721-4994-a6b5-6b3290862dc6 |*
*cinder --os-volume-api-version 3.27 attachment-delete 8daddacc-8fc8-4d2b-a738-d05deb20049f*
this is the only choice I have if I would like to unshelve vm. But this is not a good approach for production envs. I hope you understand me. Please feel free to ask me anything if you don't understand.
On Fri, 25 Nov 2022 at 13:20, Rajat Dhasmana <rdhasman@redhat.com> wrote:
Hi Hemant,
If your final goal is to delete the attachment entries in the cinder DB, we have attachment APIs to perform these tasks. The command useful for you is attachment list[1] and attachment delete[2]. Make sure you pass the right microversion i.e. 3.27 to be able to execute these operations.
Eg: cinder --os-volume-api-version 3.27 attachment-list
[1] https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinde... [2] https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinde...
On Fri, Nov 25, 2022 at 5:44 PM Hemant Sonawane < hemant.sonawane@itera.io> wrote:
Hello I am using wallaby release openstack and having issues with cinder volumes as once I try to delete, resize or unshelve the shelved vms the volume_attachement entries do not get deleted in cinder db and therefore the above mentioned operations fail every time. I have to delete these volume_attachement entries manually then it works. Is there any way to fix this issue ?
nova-compute logs:
cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume
Help will be really appreciated Thanks ! -- Thanks and Regards,
Hemant Sonawane
-- Thanks and Regards,
Hemant Sonawane
--
Sofía Enriquez
she/her
Software Engineer
Red Hat PnT <https://www.redhat.com>
IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>
-- Thanks and Regards,
Hemant Sonawane
Hello Rajat and Sofia, Sorry for the late response. Thanks to both of you. The logs I shared before are from nova only; we do not see anything apart from that in nova logs. Here are the full logs for your ready reference. 2022-12-07 10:52:16.189 23213 ERROR nova.volume.cinder [req-59de985e-9db2-4eb5-a82b-31c64f9758a9 14fd679363724aaf9e9f0503820c0b1e a95811c12f3449cf8730331fd3be1c69 - default default] Update attachment failed for attachment cab57e8a-9585-4ca5-9e16-2a75bbf16994. Error: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID: req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d) Code: 500: cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID: req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d) 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [req-59de985e-9db2-4eb5-a82b-31c64f9758a9 14fd679363724aaf9e9f0503820c0b1e a95811c12f3449cf8730331fd3be1c69 - default default] [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] Instance failed block device setup: cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID: req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d) 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] Traceback (most recent call last): 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/compute/manager.py", line 1976, in _prep_block_device 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] driver_block_device.attach_block_devices( 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 874, in attach_block_devices 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] _log_and_attach(device) 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 871, in _log_and_attach 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] bdm.attach(*attach_args, **attach_kwargs) 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 46, in wrapped 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] ret_val = method(obj, context, *args, **kwargs) 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 672, in attach 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] self._do_attach(context, instance, volume, volume_api, 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 657, in _do_attach 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] self._volume_attach(context, volume, connector, instance, 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 571, in _volume_attach 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] connection_info = volume_api.attachment_update( 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 396, in wrapper 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] res = method(self, ctx, *args, **kwargs) 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 447, in wrapper 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] res = method(self, ctx, attachment_id, *args, **kwargs) 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 875, in attachment_update 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] LOG.error('Update attachment failed for attachment ' 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] self.force_reraise() 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] raise self.value 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 867, in attachment_update 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] attachment_ref = cinderclient( 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/api_versions.py", line 423, in substitution 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] return method.func(obj, *args, **kwargs) 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/v3/attachments.py", line 75, in update 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] resp = self._update('/attachments/%s' % id, body) 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/base.py", line 312, in _update 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] resp, body = self.api.client.put(url, body=body, **kwargs) 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/client.py", line 220, in put 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] return self._cs_request(url, 'PUT', **kwargs) 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/client.py", line 205, in _cs_request 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] return self.request(url, method, **kwargs) 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/client.py", line 191, in request 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] raise exceptions.from_response(resp, body) 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID: req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d) 2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] On Wed, 7 Dec 2022 at 16:48, Rajat Dhasmana <rdhasman@redhat.com> wrote:
Hi Hemant,
Sorry for getting back to this so late. The problem you've described for instance actions like delete, resize or shelve not deleting the cinder attachment entries. I'm not sure about resize and shelve but deleting an instance should delete the volume attachment entry. I suggest you check nova logs for the calls it makes to cinder for attachment delete operation and see if there are any errors related to it. Ideally it should work but due to some issue, there might be failures. If you find a failure in nova logs related to attachment calls, you can go ahead to check cinder logs (api and volume) to see possible issues why the attachment wasn't getting deleted on the cinder side.
- Rajat Dhasmana
On Tue, Nov 29, 2022 at 2:07 PM Hemant Sonawane <hemant.sonawane@itera.io> wrote:
Hello Sofia,
Thank you for taking it into consideration. Do let me know if you have any questions and updates on the same.
On Mon, 28 Nov 2022 at 18:12, Sofia Enriquez <senrique@redhat.com> wrote:
Hi Hemant,
Thanks for reporting this issue on the bug tracker https://bugs.launchpad.net/cinder/+bug/1998083
I did a quick search and no problems with shelving operations have been reported for at least the last two years.I'll bring this bug to the cinder bug meeting this week.
Thanks Sofia
On Fri, Nov 25, 2022 at 1:15 PM Hemant Sonawane < hemant.sonawane@itera.io> wrote:
Hi Rajat, It's not about deleting attachments entries but the normal operations from horizon or via cli does not work because of that. So it really needs to be fixed to perform resize, shelve unshelve operations.
Here are the detailed attachment entries you can see for the shelved instance.
+--------------------------------------+--------------------------------------+--------------------------+---------------------------------- *----+---------------+-----------------------------------------**│·*
*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·*
*| id | volume_id | attached_host | instance_uuid | attach_status | connector │·*
*
| │·*
*+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·*
*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·*
*| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | nfv1compute43.nfv1.o2.cz <http://nfv1compute43.nfv1.o2.cz> | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | attached | {"platform": "x86_64", "os_type": "linux│·*
*", "ip": "10.42.168.87", "host": "nfv1compute43.nfv1.o2.cz <http://nfv1compute43.nfv1.o2.cz>", "multipath": false, "do_local_attach": false, "system uuid": "65917e4f-c8c4-a2af-ec11-fe353e13f4dd", "mountpoint": "/dev/vda"} | │·*
*| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a | NULL | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | reserved | NULL │·*
*
| │·*
*+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·*
*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·*
*2 rows in set (0.00 sec) *
for e.g if I would like to unshelve this instance it wont work as it has a duplicate entry in cinder db for the attachment. So i have to delete it manually from db or via cli
*root@master01:/home/hemant# cinder --os-volume-api-version 3.27 attachment-list --all | grep 67ea3a39-78b8-4d04-a280-166acdc90b8a │·*
*| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | attached | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | │·*
*| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a** | reserved | 9266a2d7-9721-4994-a6b5-6b3290862dc6 |*
*cinder --os-volume-api-version 3.27 attachment-delete 8daddacc-8fc8-4d2b-a738-d05deb20049f*
this is the only choice I have if I would like to unshelve vm. But this is not a good approach for production envs. I hope you understand me. Please feel free to ask me anything if you don't understand.
On Fri, 25 Nov 2022 at 13:20, Rajat Dhasmana <rdhasman@redhat.com> wrote:
Hi Hemant,
If your final goal is to delete the attachment entries in the cinder DB, we have attachment APIs to perform these tasks. The command useful for you is attachment list[1] and attachment delete[2]. Make sure you pass the right microversion i.e. 3.27 to be able to execute these operations.
Eg: cinder --os-volume-api-version 3.27 attachment-list
[1] https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinde... [2] https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinde...
On Fri, Nov 25, 2022 at 5:44 PM Hemant Sonawane < hemant.sonawane@itera.io> wrote:
Hello I am using wallaby release openstack and having issues with cinder volumes as once I try to delete, resize or unshelve the shelved vms the volume_attachement entries do not get deleted in cinder db and therefore the above mentioned operations fail every time. I have to delete these volume_attachement entries manually then it works. Is there any way to fix this issue ?
nova-compute logs:
cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume
Help will be really appreciated Thanks ! -- Thanks and Regards,
Hemant Sonawane
-- Thanks and Regards,
Hemant Sonawane
--
Sofía Enriquez
she/her
Software Engineer
Red Hat PnT <https://www.redhat.com>
IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>
-- Thanks and Regards,
Hemant Sonawane
-- Thanks and Regards, Hemant Sonawane
participants (3)
-
Hemant Sonawane
-
Rajat Dhasmana
-
Sofia Enriquez