<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Gorka, <div class="">Thank you very much for your help, we checked that option but before testing it we worked on the following (there is another message in the list where we have discussed it and you point it out):</div><div class="">This was the solution proposed by my colleague Sebastian: </div><div class=""><br class=""></div><div class="">> <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">We restarted the "cinder-volume" and "cinder-scheduler" services with "debug=True", got back the same debug message:</span><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">><span style="font-family: monospace;" class="">2021-05-15 23:15:27.091 31 DEBUG cinder.volume.drivers.rbd [req-f43e30ae-2bdc-4690-9c1b-3e58081fdc9e - - - - -] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). _do_conn /usr/lib/python3.6/site-packages/cinder/volume/drivers/rbd.py:431</span></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">>Then, I had a look at the docs looking for "timeout" configuration options:<br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">><a href="https://docs.openstack.org/cinder/train/configuration/block-storage/drivers/ceph-rbd-volume-driver.html#driver-options" class="">https://docs.openstack.org/cinder/train/configuration/block-storage/drivers/ceph-rbd-volume-driver.html#driver-options</a></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">>"rados_connect_timeout = -1; (Integer) Timeout value (in seconds) used when connecting to ceph cluster. If value < 0, no timeout is set and default librados value is used."</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">>I added it to the "cinder.conf" file for the "cinder-volume" service with: "rados_connect_timeout=15".</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">>Before this change the "cinder-volume" logs ended with this message:</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">><span style="font-family: monospace;" class="">2021-05-15 23:02:48.821 31 INFO cinder.volume.manager [req-6e8f9f46-ee34-4925-9fc8-dea8729d0d93 - - - - -] Starting volume driver RBDDriver (1.2.0)</span></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">>After the change:</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">><span style="font-family: monospace;" class="">2021-05-15 23:02:48.821 31 INFO cinder.volume.manager [req-6e8f9f46-ee34-4925-9fc8-dea8729d0d93 - - - - -] Starting volume driver RBDDriver (1.2.0)<br class=""></span>><span style="font-family: monospace;" class="">2021-05-15 23:04:23.180 31 INFO cinder.volume.manager [req-6e8f9f46-ee34-4925-9fc8-dea8729d0d93 - - - - -] Driver initialization completed successfully.<br class=""></span>><span style="font-family: monospace;" class="">2021-05-15 23:04:23.190 31 INFO cinder.manager [req-6e8f9f46-ee34-4925-9fc8-dea8729d0d93 - - - - -] Initiating service 12 cleanup<br class=""></span>><span style="font-family: monospace;" class="">2021-05-15 23:04:23.196 31 INFO cinder.manager [req-6e8f9f46-ee34-4925-9fc8-dea8729d0d93 - - - - -] Service 12 cleanup completed.<br class=""></span>><span style="font-family: monospace;" class="">2021-05-15 23:04:23.315 31 INFO cinder.volume.manager [req-6e8f9f46-ee34-4925-9fc8-dea8729d0d93 - - - - -] Initializing RPC dependent components of volume driver RBDDriver (1.2.0)<br class=""></span>><span style="font-family: monospace;" class="">2021-05-15 23:05:10.381 31 INFO cinder.volume.manager [req-6e8f9f46-ee34-4925-9fc8-dea8729d0d93 - - - - -] Driver post RPC initialization completed successfully.</span></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">>And now the service is reported as "up" in "openstack volume service list" and we can successfully create Ceph volumes now. Many will do more validation tests today to confirm.<br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">>So it looks like the "cinder-volume" service didn't start up properly in the first place and that's why the service was "down”.</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">Kind regards.</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class="">On 18 May 2021, at 17:49, Gorka Eguileor <<a href="mailto:geguileo@redhat.com" class="">geguileo@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">On 12/05, ManuParra wrote:<br class=""><blockquote type="cite" class="">Hi, we have faced some problems when creating volumes to add to VMs, to see what was happening I activated the Debug=True mode of Cinder in the cinder.conf file.<br class="">I see that when I try to create a new volume I get the following in the log:<br class=""><br class="">"DEBUG cinder.volume.drivers.rbd connecting to (conf=/etc/ceph/ceph.conf, timeout=-1) _do_conn /usr/lib/python3.6/site-packages/cinder/volume/drivers/rbd.py:431”<br class=""><br class="">I’m using OpenStack Train and Ceph Octopus.<br class="">When I check with openstack volume service list<br class=""><br class="">+------------------+----------------------+------+---------+-------+----------------------------+<br class="">| Binary | Host | Zone | Status | State | Updated At |<br class="">+------------------+----------------------+------+---------+-------+----------------------------+<br class="">| cinder-scheduler | spsrc-controller-1 | nova | enabled | up | 2021-05-11T10:06:39.000000 |<br class="">| cinder-scheduler | spsrc-controller-2 | nova | enabled | up | 2021-05-11T10:06:47.000000 |<br class="">| cinder-scheduler | spsrc-controller-3 | nova | enabled | up | 2021-05-11T10:06:39.000000 |<br class="">| cinder-volume | rbd:volumes@ceph-rbd | nova | enabled | down | 2021-04-11T10:48:42.000000 |<br class="">| cinder-backup | spsrc-mon-2 | nova | enabled | up | 2021-05-11T10:06:47.000000 |<br class="">| cinder-backup | spsrc-mon-1 | nova | enabled | up | 2021-05-11T10:06:44.000000 |<br class="">| cinder-backup | spsrc-mon-3 | nova | enabled | up | 2021-05-11T10:06:47.000000 |<br class="">+------------------+----------------------+------+---------+-------+——————————————+<br class=""><br class="">So cinder-volume is Down,<br class=""><br class="">I compare "cinder-backup" Ceph config with "cinder-volume", and they are equal! so why only one of them works?<br class="">diff /etc/kolla/cinder-backup/ceph.conf /etc/kolla/cinder-volume/ceph.conf<br class=""><br class="">I go inside the "cinder_volume" container<br class="">docker exec -it cinder_volume /bin/bash<br class=""><br class="">Try listing cinder volumes, works!<br class="">rbd -p cinder.volumes --id cinder -k /etc/ceph/ceph.client.cinder.keyring ls<br class=""><br class="">Any Ideas. Kind regards.<br class=""></blockquote><br class="">Hi,<br class=""><br class="">Cinder volume could be down because the stats polling is taking too<br class="">long.<br class=""><br class="">If that's the case, then you can set:<br class=""><br class=""> rbd_exclusive_cinder_pool = true<br class=""><br class="">in your driver's section in cinder.conf to fix it.<br class=""><br class="">Cheers,<br class="">Gorka.<br class=""><br class=""></div></div></blockquote></div><br class=""></div></body></html>