Support to allow only boot from cinder volumes
Hey folks, I am trying to see if there are any ways to instruct Openstack to prevent the usage of local storage/ephemeral disks. There are cases where : - I don't want the added complexity of Ceph. - I don't want the added hassle of using local volumes/nfs/shared storage on the computes directly. In an ideal world, creating a VM would mean that you always have a boot-from-volume with the volume being in your chosen backend. I've seen this spec : https://blueprints.launchpad.net/nova/+spec/flavor-root-disk-none But it doesn't seem to have survived the Ocata cycle. Any thoughts? Thanks!
On Sat, 2 Apr 2022 at 16:00, Laurent Dumont <laurentfdumont@gmail.com> wrote:
Hey folks,
Hi Laurent,
I am trying to see if there are any ways to instruct Openstack to prevent the usage of local storage/ephemeral disks.
There are cases where :
I don't want the added complexity of Ceph. I don't want the added hassle of using local volumes/nfs/shared storage on the computes directly.
In an ideal world, creating a VM would mean that you always have a boot-from-volume with the volume being in your chosen backend.
I've seen this spec : https://blueprints.launchpad.net/nova/+spec/flavor-root-disk-none
But it doesn't seem to have survived the Ocata cycle.
Any thoughts?
You can set the root and ephemeral disk sizes to 0 which means the flavor is not usable without a volume. This way you are forced to use Cinder and what it offers. -yoctozepto
On Sun, 3 Apr 2022 at 11:47, Radosław Piliszek <radoslaw.piliszek@gmail.com> wrote:
On Sat, 2 Apr 2022 at 16:00, Laurent Dumont <laurentfdumont@gmail.com> wrote:
Hey folks,
Hi Laurent,
I am trying to see if there are any ways to instruct Openstack to prevent the usage of local storage/ephemeral disks.
There are cases where :
I don't want the added complexity of Ceph. I don't want the added hassle of using local volumes/nfs/shared storage on the computes directly.
In an ideal world, creating a VM would mean that you always have a boot-from-volume with the volume being in your chosen backend.
I've seen this spec : https://blueprints.launchpad.net/nova/+spec/flavor-root-disk-none
But it doesn't seem to have survived the Ocata cycle.
Any thoughts?
You can set the root and ephemeral disk sizes to 0 which means the flavor is not usable without a volume.
Sorry, I somehow forgot to write the actual workaround. There should be this sentence in here: And then, make the nova's local disk store readonly / with a small quota.
This way you are forced to use Cinder and what it offers.
-yoctozepto
Got it! By quota, do you mean reserved_host_disk_mb in nova.conf? I could make the /var/lib/nova/instances RO but I am not sure how that would impact config drive that are created locally (since I dont have ceph) Just to be clear on the behavior, this means that boot-from-image requests would fail? Looking at nova.conf, I can disable the number of local-disks supported, but this doesn't act as a behavior change when the requests are made. I assume, from what I now know, that there is no mechanism to default/transform a request to BFV. # A negative number means unlimited. Setting max_local_block_devices# to 0 means that any request that attempts to create a local disk# will fail. This option is meant to limit the number of local discs# (so root local disc that is the result of --image being used, and# any other ephemeral and swap disks). 0 does not mean that images# will be automatically converted to volumes and boot instances from# volumes - it just means that all requests that attempt to create a# local disk will fail.## Possible values:## * 0: Creating a local disk is not allowed.# * Negative number: Allows unlimited number of local discs.# * Positive number: Allows only these many number of local discs.# (Default value is 3).# (integer value)#max_local_block_devices = 3 On Sun, Apr 3, 2022 at 5:54 AM Radosław Piliszek < radoslaw.piliszek@gmail.com> wrote:
On Sun, 3 Apr 2022 at 11:47, Radosław Piliszek <radoslaw.piliszek@gmail.com> wrote:
On Sat, 2 Apr 2022 at 16:00, Laurent Dumont <laurentfdumont@gmail.com>
Hey folks,
Hi Laurent,
I am trying to see if there are any ways to instruct Openstack to
wrote: prevent the usage of local storage/ephemeral disks.
There are cases where :
I don't want the added complexity of Ceph. I don't want the added hassle of using local volumes/nfs/shared
storage on the computes directly.
In an ideal world, creating a VM would mean that you always have a
boot-from-volume with the volume being in your chosen backend.
I've seen this spec :
https://blueprints.launchpad.net/nova/+spec/flavor-root-disk-none
But it doesn't seem to have survived the Ocata cycle.
Any thoughts?
You can set the root and ephemeral disk sizes to 0 which means the flavor is not usable without a volume.
Sorry, I somehow forgot to write the actual workaround. There should be this sentence in here: And then, make the nova's local disk store readonly / with a small quota.
This way you are forced to use Cinder and what it offers.
-yoctozepto
On Sun, 3 Apr 2022 at 12:46, Laurent Dumont <laurentfdumont@gmail.com> wrote:
Got it!
By quota, do you mean reserved_host_disk_mb in nova.conf? I could make the /var/lib/nova/instances RO but I am not sure how that would impact config drive that are created locally (since I dont have ceph)
I meant the filesystem quota. And yes, this affects config drives. Unfortunately, the error message from nova might be confusing.
Just to be clear on the behavior, this means that boot-from-image requests would fail?
Looking at nova.conf, I can disable the number of local-disks supported, but this doesn't act as a behavior change when the requests are made.
I assume, from what I now know, that there is no mechanism to default/transform a request to BFV.
# A negative number means unlimited. Setting max_local_block_devices # to 0 means that any request that attempts to create a local disk # will fail. This option is meant to limit the number of local discs # (so root local disc that is the result of --image being used, and # any other ephemeral and swap disks). 0 does not mean that images # will be automatically converted to volumes and boot instances from # volumes - it just means that all requests that attempt to create a # local disk will fail. # # Possible values: # # * 0: Creating a local disk is not allowed. # * Negative number: Allows unlimited number of local discs. # * Positive number: Allows only these many number of local discs. # (Default value is 3). # (integer value) #max_local_block_devices = 3
It seems this is actually the best approach (the error message now makes sense). Also confirmed by the faq - https://opendev.org/openstack/nova/src/commit/b0851b0e9c82446aec2ea031751476... -yoctozepto
Bummer :( We have a couple of use cases where I would like this to be transparent and things to automagically-unicorn and rainbows ;) Thanks for the insight Radoslaw! On Sun, Apr 3, 2022 at 8:12 AM Radosław Piliszek < radoslaw.piliszek@gmail.com> wrote:
On Sun, 3 Apr 2022 at 12:46, Laurent Dumont <laurentfdumont@gmail.com> wrote:
Got it!
By quota, do you mean reserved_host_disk_mb in nova.conf? I could make
the /var/lib/nova/instances RO but I am not sure how that would impact config drive that are created locally (since I dont have ceph)
I meant the filesystem quota. And yes, this affects config drives. Unfortunately, the error message from nova might be confusing.
Just to be clear on the behavior, this means that boot-from-image requests would fail?
Looking at nova.conf, I can disable the number of local-disks supported, but this doesn't act as a behavior change when the requests are made.
I assume, from what I now know, that there is no mechanism to default/transform a request to BFV.
# A negative number means unlimited. Setting max_local_block_devices # to 0 means that any request that attempts to create a local disk # will fail. This option is meant to limit the number of local discs # (so root local disc that is the result of --image being used, and # any other ephemeral and swap disks). 0 does not mean that images # will be automatically converted to volumes and boot instances from # volumes - it just means that all requests that attempt to create a # local disk will fail. # # Possible values: # # * 0: Creating a local disk is not allowed. # * Negative number: Allows unlimited number of local discs. # * Positive number: Allows only these many number of local discs. # (Default value is 3). # (integer value) #max_local_block_devices = 3
It seems this is actually the best approach (the error message now makes sense). Also confirmed by the faq -
https://opendev.org/openstack/nova/src/commit/b0851b0e9c82446aec2ea031751476...
-yoctozepto
Another way to make it transparent is using the RBD image backend in Nova and thus the instances look like they are running on local disk but is spawned on for example Ceph, however that assumes you have such a backend. In the future, which I’ve wanted for a long time, is a images backend in Nova that simply is a proxy to calling Cinder and getting a volume, that way it would be volumes in both cases but that’s a lot of work and edge cases that might need to be tuned. On 4 Apr 2022, at 00:28, Laurent Dumont <laurentfdumont@gmail.com<mailto:laurentfdumont@gmail.com>> wrote: Bummer :( We have a couple of use cases where I would like this to be transparent and things to automagically-unicorn and rainbows ;) Thanks for the insight Radoslaw! On Sun, Apr 3, 2022 at 8:12 AM Radosław Piliszek <radoslaw.piliszek@gmail.com<mailto:radoslaw.piliszek@gmail.com>> wrote: On Sun, 3 Apr 2022 at 12:46, Laurent Dumont <laurentfdumont@gmail.com<mailto:laurentfdumont@gmail.com>> wrote:
Got it!
By quota, do you mean reserved_host_disk_mb in nova.conf? I could make the /var/lib/nova/instances RO but I am not sure how that would impact config drive that are created locally (since I dont have ceph)
I meant the filesystem quota. And yes, this affects config drives. Unfortunately, the error message from nova might be confusing.
Just to be clear on the behavior, this means that boot-from-image requests would fail?
Looking at nova.conf, I can disable the number of local-disks supported, but this doesn't act as a behavior change when the requests are made.
I assume, from what I now know, that there is no mechanism to default/transform a request to BFV.
# A negative number means unlimited. Setting max_local_block_devices # to 0 means that any request that attempts to create a local disk # will fail. This option is meant to limit the number of local discs # (so root local disc that is the result of --image being used, and # any other ephemeral and swap disks). 0 does not mean that images # will be automatically converted to volumes and boot instances from # volumes - it just means that all requests that attempt to create a # local disk will fail. # # Possible values: # # * 0: Creating a local disk is not allowed. # * Negative number: Allows unlimited number of local discs. # * Positive number: Allows only these many number of local discs. # (Default value is 3). # (integer value) #max_local_block_devices = 3
It seems this is actually the best approach (the error message now makes sense). Also confirmed by the faq - https://opendev.org/openstack/nova/src/commit/b0851b0e9c82446aec2ea031751476... -yoctozepto
That is usually what we do so that boot from image also end up in something more solid than local-storage / NFS backed stuff. But having Ceph is a hassle, especially if I use Swift for the glance images. On Mon, Apr 4, 2022 at 4:45 AM Tobias Urdin <tobias.urdin@binero.com> wrote:
Another way to make it transparent is using the RBD image backend in Nova and thus the instances look like they are running on local disk but is spawned on for example Ceph, however that assumes you have such a backend.
In the future, which I’ve wanted for a long time, is a images backend in Nova that simply is a proxy to calling Cinder and getting a volume, that way it would be volumes in both cases but that’s a lot of work and edge cases that might need to be tuned.
On 4 Apr 2022, at 00:28, Laurent Dumont <laurentfdumont@gmail.com> wrote:
Bummer :(
We have a couple of use cases where I would like this to be transparent and things to automagically-unicorn and rainbows ;) Thanks for the insight Radoslaw!
On Sun, Apr 3, 2022 at 8:12 AM Radosław Piliszek < radoslaw.piliszek@gmail.com> wrote:
On Sun, 3 Apr 2022 at 12:46, Laurent Dumont <laurentfdumont@gmail.com> wrote:
Got it!
By quota, do you mean reserved_host_disk_mb in nova.conf? I could make
the /var/lib/nova/instances RO but I am not sure how that would impact config drive that are created locally (since I dont have ceph)
I meant the filesystem quota. And yes, this affects config drives. Unfortunately, the error message from nova might be confusing.
Just to be clear on the behavior, this means that boot-from-image requests would fail?
Looking at nova.conf, I can disable the number of local-disks supported, but this doesn't act as a behavior change when the requests are made.
I assume, from what I now know, that there is no mechanism to default/transform a request to BFV.
# A negative number means unlimited. Setting max_local_block_devices # to 0 means that any request that attempts to create a local disk # will fail. This option is meant to limit the number of local discs # (so root local disc that is the result of --image being used, and # any other ephemeral and swap disks). 0 does not mean that images # will be automatically converted to volumes and boot instances from # volumes - it just means that all requests that attempt to create a # local disk will fail. # # Possible values: # # * 0: Creating a local disk is not allowed. # * Negative number: Allows unlimited number of local discs. # * Positive number: Allows only these many number of local discs. # (Default value is 3). # (integer value) #max_local_block_devices = 3
It seems this is actually the best approach (the error message now makes sense). Also confirmed by the faq -
https://opendev.org/openstack/nova/src/commit/b0851b0e9c82446aec2ea031751476...
-yoctozepto
participants (3)
-
Laurent Dumont
-
Radosław Piliszek
-
Tobias Urdin