Hi Laurent, I included the Debug=True mode for Cinder-Volumes and Cinder-Scheduler, and the result is that I now have the following in the Debug: DEBUG cinder.volume.drivers.rbd [req-a0cb90b6-ca5d-496c-9a0b-e2296f1946ca - - - - -] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). _do_conn /usr/lib/python3.6/site-packages/cinder/volume/drivers/rbd.py:431 DEBUG cinder.volume.drivers.rbd [req-a0cb90b6-ca5d-496c-9a0b-e2296f1946ca - - - - -] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). _do_conn /usr/lib/python3.6/site-packages/cinder/volume/drivers/rbd.py:431 DEBUG cinder.volume.drivers.rbd [req-a0cb90b6-ca5d-496c-9a0b-e2296f1946ca - - - - -] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). _do_conn /usr/lib/python3.6/site-packages/cinder/volume/drivers/rbd.py:431 Every time a new volume is requested cinder-volumes is called which is a ceph-rbd pool. I have restarted all cinder services on the three controller/monitor nodes I have and also restarted all ceph daemons, but I still see that when doing openstack volume service list +------------------+----------------------+------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At | +------------------+----------------------+------+---------+-------+----------------------------+ | cinder-scheduler | spsrc-contr-1 | nova | enabled | up | 2021-05-11T10:06:39.000000 | | cinder-scheduler | spsrc-contr-2 | nova | enabled | up | 2021-05-11T10:06:47.000000 | | cinder-scheduler | spsrc-contr-3 | nova | enabled | up | 2021-05-11T10:06:39.000000 | | cinder-volume | rbd:volumes@ceph-rbd | nova | enabled | down | 2021-05-11T10:48:42.000000 | | cinder-backup | spsrc-mon-2 | nova | enabled | up | 2021-05-11T10:06:47.000000 | | cinder-backup | spsrc-mon-1 | nova | enabled | up | 2021-05-11T10:06:44.000000 | | cinder-backup | spsrc-mon-3 | nova | enabled | up | 2021-05-11T10:06:47.000000 | +------------------+----------------------+------+---------+-------+----------------------------+ cinder-volume is down and cannot create new volumes to associate to a VM. Kind regards.
On 12 May 2021, at 03:43, DHilsbos@performair.com <mailto:DHilsbos@performair.com> wrote:
Is this a new cluster, or one that has been running for a while?
Did you just setup integration with Ceph?
This part: "rbd:volumes@ceph-rbd" doesn't look right to me. For me (Victoria / Nautilus) this looks like: <cinder-volume-host>:<name>.
name is configured in the cinder.conf with a [<name>] section, and enabled_backends=<name> in the [DEFAULT] section. cinder-volume-host is something that resolves to the host running openstack-cinder-volume.service.
What version of OpenStack, and what version of Ceph are you running?
Thank you,
Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com <mailto:DHilsbos@PerformAir.com> www.PerformAir.com <http://www.performair.com/>
-----Original Message----- From: ManuParra [mailto:mparra@iaa.es <mailto:mparra@iaa.es>] Sent: Tuesday, May 11, 2021 3:00 PM To: Eugen Block Cc: openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> Subject: Re: Restart cinder-volume with Ceph rdb
Thanks, I have restarted the service and I see that after a few minutes then cinder-volume service goes down again when I check it with the command openstack volume service list. The host/service that contains the cinder-volumes is rbd:volumes@ceph-rbd that is RDB in Ceph, so the problem does not come from Cinder, rather from Ceph or from the RDB (Ceph) pools that stores the volumes. I have checked Ceph and the status of everything is correct, no errors or warnings. The error I have is that cinder can’t connect to rbd:volumes@ceph-rbd. Any further suggestions? Thanks in advance. Kind regards.
On 11 May 2021, at 22:30, Eugen Block <eblock@nde.ag <mailto:eblock@nde.ag>> wrote:
Hi,
so restart the volume service;-)
systemctl restart openstack-cinder-volume.service
Zitat von ManuParra <mparra@iaa.es <mailto:mparra@iaa.es>>:
Dear OpenStack community,
I have encountered a problem a few days ago and that is that when creating new volumes with:
"openstack volume create --size 20 testmv"
the volume creation status shows an error. If I go to the error log detail it indicates:
"Schedule allocate volume: Could not find any available weighted backend".
Indeed then I go to the cinder log and it indicates:
"volume service is down - host: rbd:volumes@ceph-rbd”.
I check with:
"openstack volume service list” in which state are the services and I see that indeed this happens:
| cinder-volume | rbd:volumes@ceph-rbd | nova | enabled | down | 2021-04-29T09:48:42.000000 |
And stopped since 2021-04-29 !
I have checked Ceph (monitors,managers, osds. etc) and there are no problems with the Ceph BackEnd, everything is apparently working.
This happened after an uncontrolled outage.So my question is how do I restart only cinder-volumes (I also have cinder-backup, cinder-scheduler but they are ok).
Thank you very much in advance. Regards.