[nova] local LVM volume on compute hosts
I have installed a small test cluster using openstack-ansible (xena) and would like to use local LVM volumes for storing instance volumes - and following https://docs.openstack.org/nova/latest/configuration/config.html and (e.g. https://cloudnull.io/2017/12/nova-lvm-an-iop-love-story/) I have the libvirt section of my /etc/nova/nova.conf as below. -8<--- [libvirt] inject_partition = -2 inject_password = False inject_key = False virt_type = kvm live_migration_with_native_tls = true live_migration_scheme = tls live_migration_inbound_addr = 172.29.236.5 hw_disk_discard = ignore disk_cachemodes = none images_type = lvm images_volume_group = nova volume_clear = zero -8<—— however, the nova service continues to use cinder volumes from my storage host to store instances. Is there something else that needs to be configured to make this work, or is it obsolete?
I have installed a small test cluster using openstack-ansible (xena) and would like to use local LVM volumes for storing instance volumes - and following https://docs.openstack.org/nova/latest/configuration/config.html and (e.g. https://cloudnull.io/2017/12/nova-lvm-an-iop-love-story/) I have the libvirt section of my /etc/nova/nova.conf as below.
-8<--- [libvirt] inject_partition = -2 inject_password = False inject_key = False virt_type = kvm live_migration_with_native_tls = true live_migration_scheme = tls live_migration_inbound_addr = 172.29.236.5 hw_disk_discard = ignore disk_cachemodes = none images_type = lvm images_volume_group = nova volume_clear = zero -8<——
however, the nova service continues to use cinder volumes from my storage host to store instances. Is there something else that needs to be configured to make this work, or is it obsolete? no there is noting else you need to configure but this option is not what you think it is.
On Mon, 2022-06-06 at 15:11 +0000, Paul Harrison wrote: the images_type option contols what storage will be used for all non cinder storage. i.e. vms that are booted with out usign a boot volume. by default i belive we woudl use qcow or raw files on disk with images_type=lvm we will instead create a lvm volume for the root disk but only if you do not use the boot form volume workflow. there is no way curretnly to prevent usign boot form volume on a specific host and force only local storage directly via config. you can do this indirectly but with sideffect. effectivly if you have a set of hosts that cant or should not have acess to cinder you can create a spereate avaiablity zone. you would then set [cinder]/cross_az_attch=false. e.g. create local-only az and if the cinder backend is configured to be in a differnt az then the cross_az_attach config will prevent the vm form booting in the local-only az. https://docs.openstack.org/nova/latest/configuration/config.html#cinder.cros... so your current config will make any non boot from volume nova instance use lvm storage to provision the vm root/swap/epmeral disks but will not prevent end users requesting cinder data voluems or boot volumes via the cli/api. if the opt in to cinder stoage that is what they will recive but if they use teh default storage provided by the flaovr then it will be local.
On 2022-06 -07, at 13:25, Sean Mooney <smooney@redhat.com> wrote:
no there is noting else you need to configure but this option is not what you think it is. the images_type option contols what storage will be used for all non cinder storage. i.e. vms that are booted with out usign a boot volume.
by default i belive we woudl use qcow or raw files on disk with images_type=lvm we will instead create a lvm volume for the root disk but only if you do not use the boot form volume workflow.
there is no way curretnly to prevent usign boot form volume on a specific host and force only local storage directly via config.
you can do this indirectly but with sideffect.
effectivly if you have a set of hosts that cant or should not have acess to cinder you can create a spereate avaiablity zone. you would then set [cinder]/cross_az_attch=false. e.g. create local-only az and if the cinder backend is configured to be in a differnt az then the cross_az_attach config will prevent the vm form booting in the local-only az. https://docs.openstack.org/nova/latest/configuration/config.html#cinder.cros... <https://docs.openstack.org/nova/latest/configuration/config.html#cinder.cross_az_attach>
so your current config will make any non boot from volume nova instance use lvm storage to provision the vm root/swap/epmeral disks but will not prevent end users requesting cinder data voluems or boot volumes via the cli/api. if the opt in to cinder stoage that is what they will recive but if they use teh default storage provided by the flaovr then it will be local.
thanks for the explanation - it is a shame that there is not a more direct way in config to force local storage - looks like https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools <https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools> has never got enough votes for implementation.
On Wed, 2022-06-08 at 07:26 +0000, Paul Harrison wrote:
On 2022-06 -07, at 13:25, Sean Mooney <smooney@redhat.com> wrote:
no there is noting else you need to configure but this option is not what you think it is. the images_type option contols what storage will be used for all non cinder storage. i.e. vms that are booted with out usign a boot volume.
by default i belive we woudl use qcow or raw files on disk with images_type=lvm we will instead create a lvm volume for the root disk but only if you do not use the boot form volume workflow.
there is no way curretnly to prevent usign boot form volume on a specific host and force only local storage directly via config.
you can do this indirectly but with sideffect.
effectivly if you have a set of hosts that cant or should not have acess to cinder you can create a spereate avaiablity zone. you would then set [cinder]/cross_az_attch=false. e.g. create local-only az and if the cinder backend is configured to be in a differnt az then the cross_az_attach config will prevent the vm form booting in the local-only az. https://docs.openstack.org/nova/latest/configuration/config.html#cinder.cros... <https://docs.openstack.org/nova/latest/configuration/config.html#cinder.cross_az_attach>
so your current config will make any non boot from volume nova instance use lvm storage to provision the vm root/swap/epmeral disks but will not prevent end users requesting cinder data voluems or boot volumes via the cli/api. if the opt in to cinder stoage that is what they will recive but if they use teh default storage provided by the flaovr then it will be local.
thanks for the explanation - it is a shame that there is not a more direct way in config to force local storage - looks like https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools <https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools> has never got enough votes for implementation.
thats not what your looking for really, libvirt has a way to create pools of storage locally but nova has its own way to do it itslef. so we never used libvirts way of managing storage pools becasuse in the end it did not really provide any advantage over what we alreay had. the libvirt driver can use local lvm storage or qcow/raw files but you cannot force people to use local storage without breaking interoperablity. if they request to use cinder storage that request will be fulfilled if posible or the vm will fail to boot. by default nova always uses local stroage unless you config the image backedn to rbd/ceph or the end user request boot form volume its extra work for user to boot form volume so normally they do not do this. if your users are explcitly going out of there way to not use local storage and instead are using the boot form volume flow there might be a reason for that worflow wise. nova will never use or create a cinder volume if the user does not ask for one. if you really want to prevent them more directly and interoperablity is not a concern you can alter novas rbac policy. https://github.com/openstack/nova/blob/master/nova/policies/servers.py#L254-... policy.DocumentedRuleDefault( name=SERVERS % 'create:attach_volume', check_str=base.PROJECT_MEMBER, description="Create a server with the requested volume attached to it", operations=[ { 'method': 'POST', 'path': '/servers' } ], scope_types=['project']), unfortunetly i belive that policy applies to both boot form cinder root volume and boot form local storage with a secondary data volume. your other option to acive your goal is to write custom midelware like this https://github.com/openstack/nova/blob/b5029890c1c5b1b5153c9ca2fc9a8ea2437f6... and then enable it via api-paste.ini so you declare the middelware with the fully qualifed python calsse name https://github.com/openstack/nova/blob/b5029890c1c5b1b5153c9ca2fc9a8ea2437f6... https://github.com/openstack/nova/blob/b5029890c1c5b1b5153c9ca2fc9a8ea2437f6... then you add it to the pipeline https://github.com/openstack/nova/blob/b5029890c1c5b1b5153c9ca2fc9a8ea2437f6... so if you want to block boot form volume but not block boot form local disk with an attached data volume you can do that with middelware. but in general i think it would be good to talk to your uses and understand why they are going through the extra effort to create boot form volume guests instead of just use the local storage which as i said above is the default for server create.
On 2022-06 -08, at 08:26, Paul Harrison <paul.harrison@manchester.ac.uk> wrote:
On 2022-06 -07, at 13:25, Sean Mooney <smooney@redhat.com <mailto:smooney@redhat.com>> wrote:
no there is noting else you need to configure but this option is not what you think it is. the images_type option contols what storage will be used for all non cinder storage. i.e. vms that are booted with out usign a boot volume.
so your current config will make any non boot from volume nova instance use lvm storage to provision the vm root/swap/epmeral disks but will not prevent end users requesting cinder data voluems or boot volumes via the cli/api. if the opt in to cinder stoage that is what they will recive but if they use teh default storage provided by the flaovr then it will be local.
thanks for the explanation - it is a shame that there is not a more direct way in config to force local storage - looks like https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools <https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools> has never got enough votes for implementation.
I have discovered the reason why my original setup was not working (it would fail if I tried to force in the GUI, by specifying “no” to "create new volume”) - I think it was failing here https://opendev.org/openstack/nova/src/commit/d86916360858daa06164ebc0d012b7... as the volume group device does not appear when there are no logical volumes in the group (in Centos 8 stream at least). So I forced the creation of that device by adding a dummy logical volume. Anyway, the situation is now that I can create instances that will use compute node local LVM storage, if I start them from the GUI, but not from the command line. I had a look at what the GUI sends to the /api/nova/servers/ endpoint {"availability_zone":"nova","config_drive":false,"user_data":"","disk_config":"AUTO","instance_count":1,"name":"fromgui","scheduler_hints":{},"security_groups":["48648c10-ce91-4916-9347-d88fbdba9ce6"],"create_volume_default":true,"hide_create_volume":false,"source_id":"b2a3ca46-8b0b-4748-863f-6f3e11301872","flavor_id":"da28d141-3d05-4f0f-a188-229352ccf0a3","nics":[{"net-id":"5e7a171a-ceda-4051-abe6-0496e2e8e154","v4-fixed-ip":""}],"key_name":"cloud”} However, I have not been able to find the set of command line switches for "openstack server create" that achieve the same effect - if someone knows, I would be grateful. Thanks, Paul. p.s. I was not really able to match up what the GUI sends with the API spec either. https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#cr...
On Fri, 2022-06-10 at 13:13 +0000, Paul Harrison wrote:
On 2022-06 -08, at 08:26, Paul Harrison <paul.harrison@manchester.ac.uk> wrote:
On 2022-06 -07, at 13:25, Sean Mooney <smooney@redhat.com <mailto:smooney@redhat.com>> wrote:
no there is noting else you need to configure but this option is not what you think it is. the images_type option contols what storage will be used for all non cinder storage. i.e. vms that are booted with out usign a boot volume.
so your current config will make any non boot from volume nova instance use lvm storage to provision the vm root/swap/epmeral disks but will not prevent end users requesting cinder data voluems or boot volumes via the cli/api. if the opt in to cinder stoage that is what they will recive but if they use teh default storage provided by the flaovr then it will be local.
thanks for the explanation - it is a shame that there is not a more direct way in config to force local storage - looks like https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools <https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools> has never got enough votes for implementation.
I have discovered the reason why my original setup was not working (it would fail if I tried to force in the GUI, by specifying “no” to "create new volume”) - I think it was failing here
https://opendev.org/openstack/nova/src/commit/d86916360858daa06164ebc0d012b7...
as the volume group device does not appear when there are no logical volumes in the group (in Centos 8 stream at least). So I forced the creation of that device by adding a dummy logical volume.
Anyway, the situation is now that I can create instances that will use compute node local LVM storage, if I start them from the GUI, but not from the command line.
so on the commandlien to use local storage assume you have a flavor with root disk not equal to 0 is as follows openstack server create --flavor <local storage flavor> --image <my image> --network <my network> <vm name> you can try somethign like this export IMAGE_URL=https://github.com/cirros-dev/cirros/releases/download/0.5.1/ ARCH=$(uname -m) export IMAGE=cirros-0.5.1-${ARCH}-disk.img curl --fail -L -o cirros.qcow ${IMAGE_URL}/${IMAGE} image create --disk-format qcow2 --container-format bare --public --file cirros.qcow cirros openstack flavor create --public local-storage-tiny --id auto --ram 256 --disk 1 --vcpus 1 openstack network create lvm-test openstack subnet create --subnet-range 10.0.0.0/24 --network lvm-test \ --gateway 10.0.0.1 --dns-nameserver 8.8.8.8 demo-subnet openstack router create lvm-router openstack router add subnet lvm-router lvm-subnet # assuming your public external network is called public openstack router set --external-gateway public lvm-router openstack server create --flavor local-storage-tiny --image cirros --network lvm-test local-storage-vm this is losely based on https://github.com/openstack/kolla-ansible/blob/master/tools/init-runonce
I had a look at what the GUI sends to the /api/nova/servers/ endpoint
{"availability_zone":"nova","config_drive":false,"user_data":"","disk_config":"AUTO","instance_count":1,"name":"fromgui","scheduler_hints":{},"security_groups":["48648c10-ce91-4916-9347-d88fbdba9ce6"],"create_volume_default":true,"hide_create_volume":false,"source_id":"b2a3ca46-8b0b-4748-863f-6f3e11301872","flavor_id":"da28d141-3d05-4f0f-a188-229352ccf0a3","nics":[{"net-id":"5e7a171a-ceda-4051-abe6-0496e2e8e154","v4-fixed-ip":""}],"key_name":"cloud”}
However, I have not been able to find the set of command line switches for "openstack server create" that achieve the same effect - if someone knows, I would be grateful.
Thanks, Paul.
p.s. I was not really able to match up what the GUI sends with the API spec either. https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#cr...
On 2022-06 -10, at 14:13, Paul Harrison <paul.harrison@manchester.ac.uk> wrote:
On 2022-06 -08, at 08:26, Paul Harrison <paul.harrison@manchester.ac.uk <mailto:paul.harrison@manchester.ac.uk>> wrote:
On 2022-06 -07, at 13:25, Sean Mooney <smooney@redhat.com <mailto:smooney@redhat.com>> wrote:
no there is noting else you need to configure but this option is not what you think it is. the images_type option contols what storage will be used for all non cinder storage. i.e. vms that are booted with out usign a boot volume.
so your current config will make any non boot from volume nova instance use lvm storage to provision the vm root/swap/epmeral disks but will not prevent end users requesting cinder data voluems or boot volumes via the cli/api. if the opt in to cinder stoage that is what they will recive but if they use teh default storage provided by the flaovr then it will be local.
thanks for the explanation - it is a shame that there is not a more direct way in config to force local storage - looks like https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools <https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools> has never got enough votes for implementation.
I have discovered the reason why my original setup was not working (it would fail if I tried to force in the GUI, by specifying “no” to "create new volume”) - I think it was failing here
https://opendev.org/openstack/nova/src/commit/d86916360858daa06164ebc0d012b7... <https://opendev.org/openstack/nova/src/commit/d86916360858daa06164ebc0d012b78d19ae6497/nova/virt/libvirt/imagebackend.py#L722>
as the volume group device does not appear when there are no logical volumes in the group (in Centos 8 stream at least). So I forced the creation of that device by adding a dummy logical volume.
Anyway, the situation is now that I can create instances that will use compute node local LVM storage, if I start them from the GUI, but not from the command line.
I had a look at what the GUI sends to the /api/nova/servers/ endpoint
{"availability_zone":"nova","config_drive":false,"user_data":"","disk_config":"AUTO","instance_count":1,"name":"fromgui","scheduler_hints":{},"security_groups":["48648c10-ce91-4916-9347-d88fbdba9ce6"],"create_volume_default":true,"hide_create_volume":false,"source_id":"b2a3ca46-8b0b-4748-863f-6f3e11301872","flavor_id":"da28d141-3d05-4f0f-a188-229352ccf0a3","nics":[{"net-id":"5e7a171a-ceda-4051-abe6-0496e2e8e154","v4-fixed-ip":""}],"key_name":"cloud”}
However, I have not been able to find the set of command line switches for "openstack server create" that achieve the same effect - if someone knows, I would be grateful.
Thanks, Paul.
p.s. I was not really able to match up what the GUI sends with the API spec either. https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#cr... <https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#create-server>
Have just realised, taking the hint from the GUI - if I do not try to force anything disk/volume related on the CLI, it does what I want!
On Fri, 2022-06-10 at 14:38 +0000, Paul Harrison wrote:
On 2022-06 -10, at 14:13, Paul Harrison <paul.harrison@manchester.ac.uk> wrote:
On 2022-06 -08, at 08:26, Paul Harrison <paul.harrison@manchester.ac.uk <mailto:paul.harrison@manchester.ac.uk>> wrote:
On 2022-06 -07, at 13:25, Sean Mooney <smooney@redhat.com <mailto:smooney@redhat.com>> wrote:
no there is noting else you need to configure but this option is not what you think it is. the images_type option contols what storage will be used for all non cinder storage. i.e. vms that are booted with out usign a boot volume.
so your current config will make any non boot from volume nova instance use lvm storage to provision the vm root/swap/epmeral disks but will not prevent end users requesting cinder data voluems or boot volumes via the cli/api. if the opt in to cinder stoage that is what they will recive but if they use teh default storage provided by the flaovr then it will be local.
thanks for the explanation - it is a shame that there is not a more direct way in config to force local storage - looks like https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools <https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools> has never got enough votes for implementation.
I have discovered the reason why my original setup was not working (it would fail if I tried to force in the GUI, by specifying “no” to "create new volume”) - I think it was failing here
https://opendev.org/openstack/nova/src/commit/d86916360858daa06164ebc0d012b7... <https://opendev.org/openstack/nova/src/commit/d86916360858daa06164ebc0d012b78d19ae6497/nova/virt/libvirt/imagebackend.py#L722>
as the volume group device does not appear when there are no logical volumes in the group (in Centos 8 stream at least). So I forced the creation of that device by adding a dummy logical volume.
Anyway, the situation is now that I can create instances that will use compute node local LVM storage, if I start them from the GUI, but not from the command line.
I had a look at what the GUI sends to the /api/nova/servers/ endpoint
{"availability_zone":"nova","config_drive":false,"user_data":"","disk_config":"AUTO","instance_count":1,"name":"fromgui","scheduler_hints":{},"security_groups":["48648c10-ce91-4916-9347-d88fbdba9ce6"],"create_volume_default":true,"hide_create_volume":false,"source_id":"b2a3ca46-8b0b-4748-863f-6f3e11301872","flavor_id":"da28d141-3d05-4f0f-a188-229352ccf0a3","nics":[{"net-id":"5e7a171a-ceda-4051-abe6-0496e2e8e154","v4-fixed-ip":""}],"key_name":"cloud”}
However, I have not been able to find the set of command line switches for "openstack server create" that achieve the same effect - if someone knows, I would be grateful.
Thanks, Paul.
p.s. I was not really able to match up what the GUI sends with the API spec either. https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#cr... <https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#create-server>
Have just realised, taking the hint from the GUI - if I do not try to force anything disk/volume related on the CLI, it does what I want!
yep the default behavior is what you were after. unfortunetly horizons default is differnt form nova. im not sure if you can tweak that in the horizon config so the gui also default to local sotrage but for cli users if they dont do anything fancy they will use the local lvm storage not cinder.
participants (2)
-
Paul Harrison
-
Sean Mooney