[ops][cinder][kolla] Cinder + Ceph under high load can lead to timeouts and a failure state.
In my setup of Cinder Zed + Ceph Quincy under very high loads there is a race condition due to a low timeout by default in Openstack Kolla in Cinder.conf of rados_connect_timeout = 5
This can lead to a duplicate volume trying to be made on ceph and cinder erroring out a VM Build in Nova.
Nova: nova.exception.BuildAbortException: Build of instance a529f7d5-5a81-44f6-a1fd-93caa39b2439 aborted: Volume b476094b-075e-4dc7-b63e-5bb58a0a9229 did not finish being created even after we waited 30 seconds or 4 attempts. And its status is error.
Cinder-Volume: File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 230, in __init__\n self.volume = driver.rbd.Image(rados_ioctx,\n', ' File "rbd.pyx", line 2894, in rbd.Image.__init__\n', "rbd.Timeout: [errno 110] RBD operation timeout (error opening image b'volume-b476094b-075e-4dc7-b63e-5bb58a0a9229' at snapshot None)\n"] --- File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute\n six.reraise(c, e, tb)\n', ' File "/var/lib/kolla/venv/lib/python3.9/site-packages/six.py", line 719, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker\n rv = meth(*args, **kwargs)\n', ' File "rbd.pyx", line 698, in rbd.RBD.clone\n', 'rbd.ImageExists: [errno 17] RBD image already exists (error creating clone)\n']
--- I can confirm that the volume does exist in Ceph now but no longer being tracked in Cinder, leading to a leftover artifact that eats up resources and a failed VM build. This seems like behavior that should be accounted for in some way.
Hi,
looks like that change was committed was added quite some time ago [1], but I didn't really check in detail if there has been more work on that. But apparently, having no timeout (rados_connect_timeout = -1) also caused problems with cinder-volume. We use the cinder.conf default value which also disables the timeout, it still seems to be the case in Bobcat [2]. Maybe there are specific circumstances which require a timeout, the bug report [3] doesn't have too many details about the setup. Have you tried disabling it to verify it would work for you?
Regards, Eugen
[1] https://opendev.org/openstack/kolla-ansible/commit/29a4b1996129c97b096637969... [2] https://docs.openstack.org/cinder/latest/configuration/block-storage/samples... [3] https://bugs.launchpad.net/kolla-ansible/+bug/1676267
Zitat von Forrest Fuqua fffics@rit.edu:
In my setup of Cinder Zed + Ceph Quincy under very high loads there is a race condition due to a low timeout by default in Openstack Kolla in Cinder.conf of rados_connect_timeout = 5
This can lead to a duplicate volume trying to be made on ceph and cinder erroring out a VM Build in Nova.
Nova: nova.exception.BuildAbortException: Build of instance a529f7d5-5a81-44f6-a1fd-93caa39b2439 aborted: Volume b476094b-075e-4dc7-b63e-5bb58a0a9229 did not finish being created even after we waited 30 seconds or 4 attempts. And its status is error.
Cinder-Volume: File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 230, in __init__\n self.volume = driver.rbd.Image(rados_ioctx,\n', ' File "rbd.pyx", line 2894, in rbd.Image.__init__\n', "rbd.Timeout: [errno 110] RBD operation timeout (error opening image b'volume-b476094b-075e-4dc7-b63e-5bb58a0a9229' at snapshot None)\n"]
File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute\n six.reraise(c, e, tb)\n', ' File "/var/lib/kolla/venv/lib/python3.9/site-packages/six.py", line 719, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker\n rv = meth(*args, **kwargs)\n', ' File "rbd.pyx", line 698, in rbd.RBD.clone\n', 'rbd.ImageExists: [errno 17] RBD image already exists (error creating clone)\n']
I can confirm that the volume does exist in Ceph now but no longer being tracked in Cinder, leading to a leftover artifact that eats up resources and a failed VM build. This seems like behavior that should be accounted for in some way.
I have not tried disabling it. When things settle down I will be increasing the timeout to something like 30 seconds to 60 seconds depending on testing. I was mostly surprised that ceph still created the volume and cinder didn't account for it. so it bailed on the whole transaction.
________________________________ From: Eugen Block eblock@nde.ag Sent: Monday, November 13, 2023 3:48 AM To: openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org Subject: Re: [ops][cinder][kolla] Cinder + Ceph under high load can lead to timeouts and a failure state.
Hi,
looks like that change was committed was added quite some time ago [1], but I didn't really check in detail if there has been more work on that. But apparently, having no timeout (rados_connect_timeout = -1) also caused problems with cinder-volume. We use the cinder.conf default value which also disables the timeout, it still seems to be the case in Bobcat [2]. Maybe there are specific circumstances which require a timeout, the bug report [3] doesn't have too many details about the setup. Have you tried disabling it to verify it would work for you?
Regards, Eugen
[1] https://opendev.org/openstack/kolla-ansible/commit/29a4b1996129c97b096637969... [2] https://docs.openstack.org/cinder/latest/configuration/block-storage/samples... [3] https://bugs.launchpad.net/kolla-ansible/+bug/1676267
Zitat von Forrest Fuqua fffics@rit.edu:
In my setup of Cinder Zed + Ceph Quincy under very high loads there is a race condition due to a low timeout by default in Openstack Kolla in Cinder.conf of rados_connect_timeout = 5
This can lead to a duplicate volume trying to be made on ceph and cinder erroring out a VM Build in Nova.
Nova: nova.exception.BuildAbortException: Build of instance a529f7d5-5a81-44f6-a1fd-93caa39b2439 aborted: Volume b476094b-075e-4dc7-b63e-5bb58a0a9229 did not finish being created even after we waited 30 seconds or 4 attempts. And its status is error.
Cinder-Volume: File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 230, in __init__\n self.volume = driver.rbd.Image(rados_ioctx,\n', ' File "rbd.pyx", line 2894, in rbd.Image.__init__\n', "rbd.Timeout: [errno 110] RBD operation timeout (error opening image b'volume-b476094b-075e-4dc7-b63e-5bb58a0a9229' at snapshot None)\n"]
File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute\n six.reraise(c, e, tb)\n', ' File "/var/lib/kolla/venv/lib/python3.9/site-packages/six.py", line 719, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker\n rv = meth(*args, **kwargs)\n', ' File "rbd.pyx", line 698, in rbd.RBD.clone\n', 'rbd.ImageExists: [errno 17] RBD image already exists (error creating clone)\n']
I can confirm that the volume does exist in Ceph now but no longer being tracked in Cinder, leading to a leftover artifact that eats up resources and a failed VM build. This seems like behavior that should be accounted for in some way.
participants (2)
-
Eugen Block
-
Forrest Fuqua