How to use hosts with no storage disks
In Openstack, is it possible to create compute nodes with no hard drives and use PXE in order to boot the host's system and therefore launch instances with no local drive which is needed to boot the VM's image. If not, what's the minimum storage needed to be given to hosts in order to get a fully functional system.
On 2021-10-15 11:51:49 +0100 (+0100), A Monster wrote:
In Openstack, is it possible to create compute nodes with no hard drives and use PXE in order to boot the host's system [...]
This question is outside the scope of OpenStack itself, unless you're using another OpenStack deployment to manage the physical servers (for example TripleO has an "undercloud" which uses Ironic to manage the servers which then comprise the "overcloud" presented to users). OpenStack's services start on already booted servers, so you can in theory use any mechanism you like, including PXEboot, to boot those physical servers. I understand OpenStack Ironic is a great solution to this problem though, and can be set up entirely stand-alone with its Bifrost installer. -- Jeremy Stanley
If we break it down, I'm not sure a VM will be able to boot with no volume/root disk from an image though? I guess you could have the root VM drive all in RAM, but I don't think that Openstack understands that. On Fri, Oct 15, 2021 at 8:55 AM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2021-10-15 11:51:49 +0100 (+0100), A Monster wrote:
In Openstack, is it possible to create compute nodes with no hard drives and use PXE in order to boot the host's system [...]
This question is outside the scope of OpenStack itself, unless you're using another OpenStack deployment to manage the physical servers (for example TripleO has an "undercloud" which uses Ironic to manage the servers which then comprise the "overcloud" presented to users). OpenStack's services start on already booted servers, so you can in theory use any mechanism you like, including PXEboot, to boot those physical servers. I understand OpenStack Ironic is a great solution to this problem though, and can be set up entirely stand-alone with its Bifrost installer. -- Jeremy Stanley
On 2021-10-15 12:31:02 -0400 (-0400), Laurent Dumont wrote:
If we break it down, I'm not sure a VM will be able to boot with no volume/root disk from an image though?
I guess you could have the root VM drive all in RAM, but I don't think that Openstack understands that. [...]
Well, the question seemed to be primarily about booting the underlying hardware (compute nodes) over the network. This is actually pretty commonly done, at least for provisioning, but could certainly also be used to get enough of a kernel running to find the root disk over iSCSI or whatever. As for the virtual machines (server instances), you can boot-from-volume and use any sort of remote storage Cinder supports, right? -- Jeremy Stanley
This is something we've talked about doing at Yahoo some day. There are three separate problems to solve: 1. Diskless booting the compute node off the network. Mechanically this is possible via a number of approaches. You'd have a ramdisk with the necessary components baked in, so once the ramdisk loaded you'd be in the OS. I'm not sure if this can be fully accomplished via Ironic as yet. I'd need to ask an Ironic expert to weigh in. 2. Configuration of the compute node. Either a CI job which is aware of the compute node coming up and pushing configuration via something like Ansible, or perhaps using cloud-init with the necessary pieces loaded into a config-drive image which is provided as a part of the boot process. If we can have Ironic manage diskless booting systems then this would be a solved problem with user data. 3. VM storage could either be "local" via a large ramdisk partition (assuming you have a sufficient quantity of ram in your compute nodes), an NFS share which is mounted to the compute node, or volume backed instances. We were investigating this earlier this year and got stuck on the third problem. Local storage via ramdisk isn't really an option for us, since we already pack our compute nodes with a lot of ram, and we need that memory for the instances. NFS has issues with security, since we don't want one giant volume exported to all compute nodes due to security concerns, and a per-compute node export would need to be orchestrated. Volume backed instances seemed ideal, however we ran into some issues there, which are partially related to the block storage product we use. I'm hopeful we'll get back to this next year, a class of instance flavors booted on diskless compute nodes would allow us to offer even more cost-effective options for our customers. -James On Fri, Oct 15, 2021 at 3:54 AM A Monster <amonster369@gmail.com> wrote:
In Openstack, is it possible to create compute nodes with no hard drives and use PXE in order to boot the host's system and therefore launch instances with no local drive which is needed to boot the VM's image.
If not, what's the minimum storage needed to be given to hosts in order to get a fully functional system.
Hi, On Fri, Oct 15, 2021 at 7:10 PM James Penick <jpenick@gmail.com> wrote:
This is something we've talked about doing at Yahoo some day. There are three separate problems to solve:
1. Diskless booting the compute node off the network. Mechanically this is possible via a number of approaches. You'd have a ramdisk with the necessary components baked in, so once the ramdisk loaded you'd be in the OS. I'm not sure if this can be fully accomplished via Ironic as yet. I'd need to ask an Ironic expert to weigh in.
https://docs.openstack.org/ironic/latest/admin/ramdisk-boot.html Dmitry
2. Configuration of the compute node. Either a CI job which is aware of the compute node coming up and pushing configuration via something like Ansible, or perhaps using cloud-init with the necessary pieces loaded into a config-drive image which is provided as a part of the boot process. If we can have Ironic manage diskless booting systems then this would be a solved problem with user data. 3. VM storage could either be "local" via a large ramdisk partition (assuming you have a sufficient quantity of ram in your compute nodes), an NFS share which is mounted to the compute node, or volume backed instances.
We were investigating this earlier this year and got stuck on the third problem. Local storage via ramdisk isn't really an option for us, since we already pack our compute nodes with a lot of ram, and we need that memory for the instances. NFS has issues with security, since we don't want one giant volume exported to all compute nodes due to security concerns, and a per-compute node export would need to be orchestrated. Volume backed instances seemed ideal, however we ran into some issues there, which are partially related to the block storage product we use. I'm hopeful we'll get back to this next year, a class of instance flavors booted on diskless compute nodes would allow us to offer even more cost-effective options for our customers.
-James
On Fri, Oct 15, 2021 at 3:54 AM A Monster <amonster369@gmail.com> wrote:
In Openstack, is it possible to create compute nodes with no hard drives and use PXE in order to boot the host's system and therefore launch instances with no local drive which is needed to boot the VM's image.
If not, what's the minimum storage needed to be given to hosts in order to get a fully functional system.
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
As far as I know, ironic aims to provision bare metal machines instead of virtual machines, in my case, what I want to accomplish is to boot the host's operating system through network, and then use either a remote disk in which the image service copies the vm's image to, and then boot from that image, or if it's possible, use the ram instead of a disk for that task, and that would allow me to use diskless computer nodes (hosts). On Fri, 15 Oct 2021 at 18:06, James Penick <jpenick@gmail.com> wrote:
This is something we've talked about doing at Yahoo some day. There are three separate problems to solve:
1. Diskless booting the compute node off the network. Mechanically this is possible via a number of approaches. You'd have a ramdisk with the necessary components baked in, so once the ramdisk loaded you'd be in the OS. I'm not sure if this can be fully accomplished via Ironic as yet. I'd need to ask an Ironic expert to weigh in. 2. Configuration of the compute node. Either a CI job which is aware of the compute node coming up and pushing configuration via something like Ansible, or perhaps using cloud-init with the necessary pieces loaded into a config-drive image which is provided as a part of the boot process. If we can have Ironic manage diskless booting systems then this would be a solved problem with user data. 3. VM storage could either be "local" via a large ramdisk partition (assuming you have a sufficient quantity of ram in your compute nodes), an NFS share which is mounted to the compute node, or volume backed instances.
We were investigating this earlier this year and got stuck on the third problem. Local storage via ramdisk isn't really an option for us, since we already pack our compute nodes with a lot of ram, and we need that memory for the instances. NFS has issues with security, since we don't want one giant volume exported to all compute nodes due to security concerns, and a per-compute node export would need to be orchestrated. Volume backed instances seemed ideal, however we ran into some issues there, which are partially related to the block storage product we use. I'm hopeful we'll get back to this next year, a class of instance flavors booted on diskless compute nodes would allow us to offer even more cost-effective options for our customers.
-James
On Fri, Oct 15, 2021 at 3:54 AM A Monster <amonster369@gmail.com> wrote:
In Openstack, is it possible to create compute nodes with no hard drives and use PXE in order to boot the host's system and therefore launch instances with no local drive which is needed to boot the VM's image.
If not, what's the minimum storage needed to be given to hosts in order to get a fully functional system.
You are correct, I meant you would use Ironic to provision the compute node, which Nova would then use to provision VMs. On Fri, Oct 15, 2021 at 11:16 AM A Monster <amonster369@gmail.com> wrote:
As far as I know, ironic aims to provision bare metal machines instead of virtual machines, in my case, what I want to accomplish is to boot the host's operating system through network, and then use either a remote disk in which the image service copies the vm's image to, and then boot from that image, or if it's possible, use the ram instead of a disk for that task, and that would allow me to use diskless computer nodes (hosts).
On Fri, 15 Oct 2021 at 18:06, James Penick <jpenick@gmail.com> wrote:
This is something we've talked about doing at Yahoo some day. There are three separate problems to solve:
1. Diskless booting the compute node off the network. Mechanically this is possible via a number of approaches. You'd have a ramdisk with the necessary components baked in, so once the ramdisk loaded you'd be in the OS. I'm not sure if this can be fully accomplished via Ironic as yet. I'd need to ask an Ironic expert to weigh in. 2. Configuration of the compute node. Either a CI job which is aware of the compute node coming up and pushing configuration via something like Ansible, or perhaps using cloud-init with the necessary pieces loaded into a config-drive image which is provided as a part of the boot process. If we can have Ironic manage diskless booting systems then this would be a solved problem with user data. 3. VM storage could either be "local" via a large ramdisk partition (assuming you have a sufficient quantity of ram in your compute nodes), an NFS share which is mounted to the compute node, or volume backed instances.
We were investigating this earlier this year and got stuck on the third problem. Local storage via ramdisk isn't really an option for us, since we already pack our compute nodes with a lot of ram, and we need that memory for the instances. NFS has issues with security, since we don't want one giant volume exported to all compute nodes due to security concerns, and a per-compute node export would need to be orchestrated. Volume backed instances seemed ideal, however we ran into some issues there, which are partially related to the block storage product we use. I'm hopeful we'll get back to this next year, a class of instance flavors booted on diskless compute nodes would allow us to offer even more cost-effective options for our customers.
-James
On Fri, Oct 15, 2021 at 3:54 AM A Monster <amonster369@gmail.com> wrote:
In Openstack, is it possible to create compute nodes with no hard drives and use PXE in order to boot the host's system and therefore launch instances with no local drive which is needed to boot the VM's image.
If not, what's the minimum storage needed to be given to hosts in order to get a fully functional system.
Issue 3, as laid out below, can be addressed using Ceph RBD. Use it behind cinder & glance, and no local storage is required. Our OpenStack cluster has small OS drives, and doesn't store either volumes or images locally. Thank you, Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com From: James Penick [mailto:jpenick@gmail.com] Sent: Friday, October 15, 2021 11:18 AM To: A Monster Cc: openstack-discuss Subject: Re: How to use hosts with no storage disks You are correct, I meant you would use Ironic to provision the compute node, which Nova would then use to provision VMs. On Fri, Oct 15, 2021 at 11:16 AM A Monster <amonster369@gmail.com> wrote: As far as I know, ironic aims to provision bare metal machines instead of virtual machines, in my case, what I want to accomplish is to boot the host's operating system through network, and then use either a remote disk in which the image service copies the vm's image to, and then boot from that image, or if it's possible, use the ram instead of a disk for that task, and that would allow me to use diskless computer nodes (hosts). On Fri, 15 Oct 2021 at 18:06, James Penick <jpenick@gmail.com> wrote: This is something we've talked about doing at Yahoo some day. There are three separate problems to solve: 1. Diskless booting the compute node off the network. Mechanically this is possible via a number of approaches. You'd have a ramdisk with the necessary components baked in, so once the ramdisk loaded you'd be in the OS. I'm not sure if this can be fully accomplished via Ironic as yet. I'd need to ask an Ironic expert to weigh in. 2. Configuration of the compute node. Either a CI job which is aware of the compute node coming up and pushing configuration via something like Ansible, or perhaps using cloud-init with the necessary pieces loaded into a config-drive image which is provided as a part of the boot process. If we can have Ironic manage diskless booting systems then this would be a solved problem with user data. 3. VM storage could either be "local" via a large ramdisk partition (assuming you have a sufficient quantity of ram in your compute nodes), an NFS share which is mounted to the compute node, or volume backed instances. We were investigating this earlier this year and got stuck on the third problem. Local storage via ramdisk isn't really an option for us, since we already pack our compute nodes with a lot of ram, and we need that memory for the instances. NFS has issues with security, since we don't want one giant volume exported to all compute nodes due to security concerns, and a per-compute node export would need to be orchestrated. Volume backed instances seemed ideal, however we ran into some issues there, which are partially related to the block storage product we use. I'm hopeful we'll get back to this next year, a class of instance flavors booted on diskless compute nodes would allow us to offer even more cost-effective options for our customers. -James On Fri, Oct 15, 2021 at 3:54 AM A Monster <amonster369@gmail.com> wrote: In Openstack, is it possible to create compute nodes with no hard drives and use PXE in order to boot the host's system and therefore launch instances with no local drive which is needed to boot the VM's image. If not, what's the minimum storage needed to be given to hosts in order to get a fully functional system.
participants (6)
-
A Monster
-
DHilsbos＠performair.com
-
Dmitry Tantsur
-
James Penick
-
Jeremy Stanley
-
Laurent Dumont