[cinder][nova] force detach volume
Hey team, Is there any way to force detach a volume when the nova compute service is down? Like when the hypervisor is off, we want to detach the volume and attach it to another instance. Maybe I'm missing the way to do it, but detach action on nova side seems a synchronous task that I can't force without admin creds. Thanks for any help, Regards, Arnaud
I can partially answer myself. This is a feature, which prevent inconsistencies from nova/cinder. https://review.opendev.org/c/openstack/nova/+/768352 https://bugs.launchpad.net/nova/+bug/1909120 Now, I am wondering if multiattach could be a partial solution? Cheers, Arnaud On 20.03.25 - 13:49, Arnaud Morin wrote:
Hey team,
Is there any way to force detach a volume when the nova compute service is down? Like when the hypervisor is off, we want to detach the volume and attach it to another instance.
Maybe I'm missing the way to do it, but detach action on nova side seems a synchronous task that I can't force without admin creds.
Thanks for any help,
Regards, Arnaud
Hello, I think we usually have an evacuation api for this. Is there any reason you do not want to evacuate instance, then the volume will stay attached and usuable ? Regards On Fri, Mar 21, 2025 at 4:34 PM Arnaud Morin <arnaud.morin@gmail.com> wrote:
I can partially answer myself. This is a feature, which prevent inconsistencies from nova/cinder.
https://review.opendev.org/c/openstack/nova/+/768352 https://bugs.launchpad.net/nova/+bug/1909120
Now, I am wondering if multiattach could be a partial solution?
Cheers, Arnaud
On 20.03.25 - 13:49, Arnaud Morin wrote:
Hey team,
Is there any way to force detach a volume when the nova compute service is down? Like when the hypervisor is off, we want to detach the volume and attach it to another instance.
Maybe I'm missing the way to do it, but detach action on nova side seems a synchronous task that I can't force without admin creds.
Thanks for any help,
Regards, Arnaud
Hey, Most of our instances are having local disk. And some of them have attached volumes. Evacuation does not seem possible in that situation, am I wrong? Moreover, evacuation seems an administrator option, not beeing able to be triggered from an user action. My use case is the following: I have a region with multiple AZ, cross_az attachment is allowed for my volumes. There is no issue to attach / detach until the the compute is down. The compute could be down when an AZ is down. In that situation, a use may want to force detach the volume and re-attach it to another instances in another AZ. Regards, Arnaud On 21.03.25 - 17:00, Amit Uniyal wrote:
Hello, I think we usually have an evacuation api for this. Is there any reason you do not want to evacuate instance, then the volume will stay attached and usuable ?
Regards
On Fri, Mar 21, 2025 at 4:34 PM Arnaud Morin <arnaud.morin@gmail.com> wrote:
I can partially answer myself. This is a feature, which prevent inconsistencies from nova/cinder.
https://review.opendev.org/c/openstack/nova/+/768352 https://bugs.launchpad.net/nova/+bug/1909120
Now, I am wondering if multiattach could be a partial solution?
Cheers, Arnaud
On 20.03.25 - 13:49, Arnaud Morin wrote:
Hey team,
Is there any way to force detach a volume when the nova compute service is down? Like when the hypervisor is off, we want to detach the volume and attach it to another instance.
Maybe I'm missing the way to do it, but detach action on nova side seems a synchronous task that I can't force without admin creds.
Thanks for any help,
Regards, Arnaud
thansk for expalining unfortunately as i mentioned forcing a detach is inherintly unsafe and not something tha twe can support upstream. if your cloud is using cpeh or perhaps nfs (i.e. a backend that does not host mount a cinder volumes before attaching it to a guest) it is potientlaly safe but nova cant genericly know that that is ok to do. before we blocked user force detaching form the cinder api deleting the attachments or volumes would leave nova db in a currpted state as the nova database would still have recored fo the attachmens in the block device mapping table. so ignoring the security implications it was still unsupproted even before that changes. unfortnely without effectivly implemeting a way to do a delated detach when the comptue starts up there is no safe way to allow a standard user with just the member role to force detach a volume. On Fri, 2025-03-21 at 21:07 +0000, Arnaud Morin wrote:
Hey,
Most of our instances are having local disk. And some of them have attached volumes. Evacuation does not seem possible in that situation, am I wrong?
Moreover, evacuation seems an administrator option, not beeing able to be triggered from an user action.
My use case is the following: I have a region with multiple AZ, cross_az attachment is allowed for my volumes. There is no issue to attach / detach until the the compute is down. The compute could be down when an AZ is down. In that situation, a use may want to force detach the volume and re-attach it to another instances in another AZ.
Regards,
Arnaud
On 21.03.25 - 17:00, Amit Uniyal wrote:
Hello, I think we usually have an evacuation api for this. Is there any reason you do not want to evacuate instance, then the volume will stay attached and usuable ?
Regards
On Fri, Mar 21, 2025 at 4:34 PM Arnaud Morin <arnaud.morin@gmail.com> wrote:
I can partially answer myself. This is a feature, which prevent inconsistencies from nova/cinder.
https://review.opendev.org/c/openstack/nova/+/768352 https://bugs.launchpad.net/nova/+bug/1909120
Now, I am wondering if multiattach could be a partial solution?
Cheers, Arnaud
On 20.03.25 - 13:49, Arnaud Morin wrote:
Hey team,
Is there any way to force detach a volume when the nova compute service is down? Like when the hypervisor is off, we want to detach the volume and attach it to another instance.
Maybe I'm missing the way to do it, but detach action on nova side seems a synchronous task that I can't force without admin creds.
Thanks for any help,
Regards, Arnaud
I don’t know if it would enable you to attach the volume to a different vm, but you can change a volume‘s database status to detached and available (check out ‚openstack help volume set‘). Zitat von Arnaud Morin <arnaud.morin@gmail.com>:
Hey team,
Is there any way to force detach a volume when the nova compute service is down? Like when the hypervisor is off, we want to detach the volume and attach it to another instance.
Maybe I'm missing the way to do it, but detach action on nova side seems a synchronous task that I can't force without admin creds.
Thanks for any help,
Regards, Arnaud
so in genreal no. for security reasons we do not allow humans, even admins to detach volume via the cidner api. https://security.openstack.org/ossa/OSSA-2023-003.html we have locked down access to deletating a volume attachment requests that have a service user token. https://security.openstack.org/ossa/OSSA-2023-003.html#configuration-change while an admid coudl impersonate nova with curl you intentially cannot craft a request like this with the openstack commandlien client. if you were to do that and not also manually clean the compute host you would make the cloud vulnerable to the cve again. any combination of commands that allow a normal user to bypass this protection would be a CVE so be very careful with seculating in public about that. if you find one please report it privately https://security.openstack.org/reporting.html On Fri, 2025-03-21 at 15:02 +0000, Eugen Block wrote:
I don’t know if it would enable you to attach the volume to a different vm, but you can change a volume‘s database status to detached and available (check out ‚openstack help volume set‘).
Zitat von Arnaud Morin <arnaud.morin@gmail.com>:
Hey team,
Is there any way to force detach a volume when the nova compute service is down? Like when the hypervisor is off, we want to detach the volume and attach it to another instance. you only reall option if the hypervior is dead woudl be to evacuate the instance and then detach it after the evacuation. that still not advisable but might work for your usecase.
Maybe I'm missing the way to do it, but detach action on nova side seems a synchronous task that I can't force without admin creds.
correct we need to do clean up on the compute host in almost all cases (all excpet ceph) to prevetn unatuorised access to the volume. if that does not succesedd we will potionally leak data or currpt the volume.
Thanks for any help,
Regards, Arnaud
participants (4)
-
Amit Uniyal
-
Arnaud Morin
-
Eugen Block
-
smooney@redhat.com