<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">Hi Laurent, I included the Debug=True mode for Cinder-Volumes and Cinder-Scheduler, and the result is that I now have the following in the Debug:</div><div class=""><br class=""></div><div class=""><div class="index7 line number8 alt1"> <code class="java plain">DEBUG cinder.volume.drivers.rbd [req-a0cb90b6-ca5d-496c-9a0b-e2296f1946ca - - - - -] connecting to cinder</code><code class="java color1">@ceph</code> <code class="java plain">(conf=/etc/ceph/ceph.conf, timeout=-</code><code class="java value">1</code><code class="java plain">). _do_conn /usr/lib/python3.</code><code class="java value">6</code><code class="java plain">/site-packages/cinder/volume/drivers/rbd.py:</code><code class="java value">431</code></div><div class="alt2 line index8 number9"> <code class="java plain">DEBUG cinder.volume.drivers.rbd [req-a0cb90b6-ca5d-496c-9a0b-e2296f1946ca - - - - -] connecting to cinder</code><code class="java color1">@ceph</code> <code class="java plain">(conf=/etc/ceph/ceph.conf, timeout=-</code><code class="java value">1</code><code class="java plain">). _do_conn /usr/lib/python3.</code><code class="java value">6</code><code class="java plain">/site-packages/cinder/volume/drivers/rbd.py:</code><code class="java value">431</code></div><div class="number10 line index9 alt1"> <code class="java plain">DEBUG cinder.volume.drivers.rbd [req-a0cb90b6-ca5d-496c-9a0b-e2296f1946ca - - - - -] connecting to cinder</code><code class="java color1">@ceph</code> <code class="java plain">(conf=/etc/ceph/ceph.conf, timeout=-</code><code class="java value">1</code><code class="java plain">). _do_conn /usr/lib/python3.</code><code class="java value">6</code><code class="java plain">/site-packages/cinder/volume/drivers/r</code><code class="java plain" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">bd.py:</code><code class="java value" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">431</code></div></div><div class=""><br class=""></div><div class="">Every time a new volume is requested cinder-volumes is called which is a ceph-rbd pool.</div><div class=""><br class=""></div><div class="">I have restarted all cinder services on the three controller/monitor nodes I have and also restarted all ceph daemons, but I still see that when doing </div><div class="">openstack volume service list</div><div class=""><br class=""></div><div class=""><div class="container" title="Hint: double-click to select code"><div class="index0 line number1 alt2"><code class="java plain">+------------------+----------------------+------+---------+-------+----------------------------+</code></div><div class="line index1 alt1 number2"><code class="java plain">| Binary | Host | Zone | Status | State | Updated At |</code></div><div class="alt2 line number3 index2"><code class="java plain">+------------------+----------------------+------+---------+-------+----------------------------+</code></div><div class="index3 line number4 alt1"><code class="java plain">| cinder-scheduler | spsrc-contr-</code><code class="java value">1</code> <code class="java plain">| nova | enabled | up | </code><code class="java value">2021</code><code class="java plain">-</code><code class="java value">05</code><code class="java plain">-11T10:</code><code class="java value">06</code><code class="java plain">:</code><code class="java value">39.000000</code> <code class="java plain">|</code></div><div class="number5 line index4 alt2"><code class="java plain">| cinder-scheduler | spsrc-contr-</code><code class="java value">2</code> <code class="java plain">| nova | enabled | up | </code><code class="java value">2021</code><code class="java plain">-</code><code class="java value">05</code><code class="java plain">-11T10:</code><code class="java value">06</code><code class="java plain">:</code><code class="java value">47.000000</code> <code class="java plain">|</code></div><div class="line number6 index5 alt1"><code class="java plain">| cinder-scheduler | spsrc-contr-</code><code class="java value">3</code> <code class="java plain">| nova | enabled | up | </code><code class="java value">2021</code><code class="java plain">-</code><code class="java value">05</code><code class="java plain">-11T10:</code><code class="java value">06</code><code class="java plain">:</code><code class="java value">39.000000</code> <code class="java plain">|</code></div><div class="number7 alt2 line index6"><code class="java plain">| cinder-volume | rbd:volumes</code><code class="java color1">@ceph</code><code class="java plain">-rbd | nova | enabled | down | </code><code class="java value">2021</code><code class="java plain">-</code><code class="java value">05</code><code class="java plain">-11T10:</code><code class="java value">48</code><code class="java plain">:</code><code class="java value">42.000000</code> <code class="java plain">|</code></div><div class="index7 line number8 alt1"><code class="java plain">| cinder-backup | spsrc-mon-</code><code class="java value">2</code> <code class="java plain">| nova | enabled | up | </code><code class="java value">2021</code><code class="java plain">-</code><code class="java value">05</code><code class="java plain">-11T10:</code><code class="java value">06</code><code class="java plain">:</code><code class="java value">47.000000</code> <code class="java plain">|</code></div><div class="alt2 line index8 number9"><code class="java plain">| cinder-backup | spsrc-mon-</code><code class="java value">1</code> <code class="java plain">| nova | enabled | up | </code><code class="java value">2021</code><code class="java plain">-</code><code class="java value">05</code><code class="java plain">-11T10:</code><code class="java value">06</code><code class="java plain">:</code><code class="java value">44.000000</code> <code class="java plain">|</code></div><div class="number10 line index9 alt1"><code class="java plain">| cinder-backup | spsrc-mon-</code><code class="java value">3</code> <code class="java plain">| nova | enabled | up | </code><code class="java value">2021</code><code class="java plain">-</code><code class="java value">05</code><code class="java plain">-11T10:</code><code class="java value">06</code><code class="java plain">:</code><code class="java value">47.000000</code> <code class="java plain">|</code></div><div class="index10 line alt2 number11"><code class="java plain">+------------------+----------------------+------+---------+-------+----------------------------+</code></div></div></div><div class=""><br class=""></div><div class="">cinder-volume is down and cannot create new volumes to associate to a VM.</div><div class=""><br class=""></div><div class="">Kind regards.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 12 May 2021, at 03:43, <a href="mailto:DHilsbos@performair.com" class="">DHilsbos@performair.com</a> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Is this a new cluster, or one that has been running for a while?<br class=""><br class="">Did you just setup integration with Ceph?<br class=""><br class="">This part: "rbd:volumes@ceph-rbd" doesn't look right to me. For me (Victoria / Nautilus) this looks like: <cinder-volume-host>:<name>.<br class=""><br class="">name is configured in the cinder.conf with a [<name>] section, and enabled_backends=<name> in the [DEFAULT] section.<br class="">cinder-volume-host is something that resolves to the host running openstack-cinder-volume.service.<br class=""><br class="">What version of OpenStack, and what version of Ceph are you running?<br class=""><br class="">Thank you,<br class=""><br class="">Dominic L. Hilsbos, MBA <br class="">Vice President – Information Technology <br class="">Perform Air International Inc.<br class=""><a href="mailto:DHilsbos@PerformAir.com" class="">DHilsbos@PerformAir.com</a> <br class=""><a href="http://www.performair.com/" class="">www.PerformAir.com</a><br class=""><br class=""><br class="">-----Original Message-----<br class="">From: ManuParra [<a href="mailto:mparra@iaa.es" class="">mailto:mparra@iaa.es</a>] <br class="">Sent: Tuesday, May 11, 2021 3:00 PM<br class="">To: Eugen Block<br class="">Cc: <a href="mailto:openstack-discuss@lists.openstack.org" class="">openstack-discuss@lists.openstack.org</a><br class="">Subject: Re: Restart cinder-volume with Ceph rdb<br class=""><br class="">Thanks, I have restarted the service and I see that after a few minutes then cinder-volume service goes down again when I check it with the command openstack volume service list. <br class="">The host/service that contains the cinder-volumes is rbd:volumes@ceph-rbd that is RDB in Ceph, so the problem does not come from Cinder, rather from Ceph or from the RDB (Ceph) pools that stores the volumes. I have checked Ceph and the status of everything is correct, no errors or warnings. <br class="">The error I have is that cinder can’t connect to rbd:volumes@ceph-rbd. Any further suggestions? Thanks in advance.<br class="">Kind regards.<br class=""><br class=""><blockquote type="cite" class="">On 11 May 2021, at 22:30, Eugen Block <<a href="mailto:eblock@nde.ag" class="">eblock@nde.ag</a>> wrote:<br class=""><br class="">Hi,<br class=""><br class="">so restart the volume service;-)<br class=""><br class="">systemctl restart openstack-cinder-volume.service<br class=""><br class=""><br class="">Zitat von ManuParra <<a href="mailto:mparra@iaa.es" class="">mparra@iaa.es</a>>:<br class=""><br class=""><blockquote type="cite" class="">Dear OpenStack community,<br class=""><br class="">I have encountered a problem a few days ago and that is that when creating new volumes with:<br class=""><br class="">"openstack volume create --size 20 testmv"<br class=""><br class="">the volume creation status shows an error. If I go to the error log detail it indicates:<br class=""><br class="">"Schedule allocate volume: Could not find any available weighted backend".<br class=""><br class="">Indeed then I go to the cinder log and it indicates:<br class=""><br class="">"volume service is down - host: rbd:volumes@ceph-rbd”.<br class=""><br class="">I check with:<br class=""><br class="">"openstack volume service list” in which state are the services and I see that indeed this happens:<br class=""><br class=""><br class="">| cinder-volume | rbd:volumes@ceph-rbd | nova | enabled | down | 2021-04-29T09:48:42.000000 |<br class=""><br class="">And stopped since 2021-04-29 !<br class=""><br class="">I have checked Ceph (monitors,managers, osds. etc) and there are no problems with the Ceph BackEnd, everything is apparently working.<br class=""><br class="">This happened after an uncontrolled outage.So my question is how do I restart only cinder-volumes (I also have cinder-backup, cinder-scheduler but they are ok).<br class=""><br class="">Thank you very much in advance. Regards.<br class=""></blockquote><br class=""><br class=""><br class=""><br class=""></blockquote><br class=""><br class=""><br class=""></div></div></blockquote></div><br class=""></div></div></body></html>