Hi Michal, We had similar issue in our kolla-ansible deployment. In our case we were not able to attach volume to the VM in 1 from 5 cases and it sometimes caused also inconsistencies in DB tables. We are also running Victoria release with Ceph backend. We found out that there were many disconnects/reconnects between nova and rabbitmq because of missing heartbeats which happened every 3 minutes. If attachment of volume happened in the time of reconnection, attachment was unsuccessful. What helped in our case was adding following lines to nova.conf: [oslo_messaging_rabbit] heartbeat_timeout_threshold = 0 In your case I would check cinder logs files for messaging errors whether you do not see regular reconnects to rabbitmq. Kamil Madac On Fri, Jun 10, 2022 at 2:08 PM Michal Arbet <michal.arbet@ultimum.io> wrote:

Hi,

Cinder is from victoria, backend is ceph, steps to reproduce ? Just delete volume. I think that it could be an error not in cinder but somehow a rabbitmq issue.

:/

Michal Arbet Openstack Engineer

Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic

+420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io <https://ultimum.io/>*

LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter <https://twitter.com/ultimumtech> | Facebook <https://www.facebook.com/ultimumtechnologies/timeline>

út 7. 6. 2022 v 19:27 odesílatel Sofia Enriquez <senrique@redhat.com> napsal:

...

Hi Michal,

I do not think we have such a bug on the Launchpad platform.[1] Could you create a new bug report indicating which backend you are using, which Cinder version you have, and if the error occurs after following some steps?

Thanks in advance, Sofia

[1] https://bugs.launchpad.net/cinder

On Tue, Jun 7, 2022 at 4:28 AM Michal Arbet <michal.arbet@ultimum.io> wrote:

...

Hi,

I would like to ask if someone saw issue when volume failed to delete, log below : This is happening from time to time.

LOG : https://paste.opendev.org/show/b1jRCxuBFnnVzf64i9yV/

Thanks,

Michal Arbet Openstack Engineer

Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic

+420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io <https://ultimum.io/>*

LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter <https://twitter.com/ultimumtech> | Facebook <https://www.facebook.com/ultimumtechnologies/timeline>

--

Sofía Enriquez

she/her

Software Engineer

Red Hat PnT <https://www.redhat.com>

IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

-- Kamil Madac <https://kmadac.github.io/>

Hi Gorka, Yeah, we are using active-active cinder deployment. - Not defined host - Not defined backend_host - Defined cluster And redis-sentinel as coordinator Binary Host Zone Status State Updated At RPC Version Object Version Cluster cinder-scheduler cn36x005 nova enabled :-) 2022-06-15 05:56:49 3.12 1.38 cinder-scheduler cv36x006 nova enabled :-) 2022-06-15 05:56:54 3.12 1.38 cinder-backup cn36x005 nova enabled :-) 2022-06-15 05:56:55 2.2 1.38 cinder-backup cv36x006 nova enabled :-) 2022-06-15 05:56:49 2.2 1.38 cinder-volume cn36x005@rbd-2 nova enabled :-) 2022-06-15 05:56:50 3.16 1.38 cinder_cl1@rbd-2 cinder-volume cv36x006@rbd-2 nova enabled :-) 2022-06-15 05:56:53 3.16 1.38 cinder_cl1@rbd-2 cinder-scheduler cn36x009 nova enabled :-) 2022-06-15 05:56:50 3.12 1.38 cinder-scheduler cv36x010 nova enabled :-) 2022-06-15 05:56:57 3.12 1.38 cinder-volume cn36x005@rbd-1 nova enabled :-) 2022-06-15 05:56:54 3.16 1.38 cinder_cl1@rbd-1 cinder-volume cv36x006@rbd-1 nova enabled :-) 2022-06-15 05:56:53 3.16 1.38 cinder_cl1@rbd-1 Michal Arbet Openstack Engineer Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic +420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io <https://ultimum.io/>* LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter <https://twitter.com/ultimumtech> | Facebook <https://www.facebook.com/ultimumtechnologies/timeline> út 14. 6. 2022 v 9:38 odesílatel Gorka Eguileor <geguileo@redhat.com> napsal:

On 07/06, Michal Arbet wrote:

...

Hi,

I would like to ask if someone saw issue when volume failed to delete, log below : This is happening from time to time.

LOG : https://paste.opendev.org/show/b1jRCxuBFnnVzf64i9yV/

Thanks,

Michal Arbet Openstack Engineer

Hi,

Error seems to be happening acquiring a lock with the Redis driver in Tooz, so it must be configured to be used by Cinder as the DLM.

This is only necessary when the Cinder Volume service is configured as Active-Active. Are you running the volume service as Active-Active?

If you are not you should be able to fix the issue by commenting the backend_url configuration option in the [coordination] section of the cinder.conf file.

Cheers, Gorka.

...

Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic

+420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io <https://ultimum.io/>*

LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter <https://twitter.com/ultimumtech> | Facebook <https://www.facebook.com/ultimumtechnologies/timeline>