Hi Tony,

Thanks for your reply.

On 12/2/21 12:47 AM, Tony Liu wrote:

Here is what I would do, not validated.

• Create a network with DHCP disabled.
• Launch a VM to support PXE boot.
• Launch a VM on volume based on any image for PXE boot.
• The default virtio nic works fine with PXE boot.
• "virsh edit <instance>" to enable PXE boot on the compute node.

This isn't an option. To do "virsh edit", you need to have root access to a compute node, and that's not what I would like to do: I'd like my CI to run with a normal OpenStack account only, to make it easy for people to reuse.

• "soft reboot" VM should bring VM to PXE boot.

I prefer to do this with virsh/libvirt directly, which is much easier.

Tony

Cheers,

Thomas Goirand (zigo)

On 12/2/21 12:51 AM, Clark Boylan wrote:

I'm not super familiar with all the details but I believe ironic does something similar in their devstack plugin [2]. They do this so they can create fake baremetal instances that are actually VMs that ironic manages. But a similar setup will probably work for your use cases if PXE is what you are starting with.

[2] https://opendev.org/openstack/ironic/src/branch/master/devstack/lib/ironic#L...

Hi,

Thanks Clark for your answer.

I've read the script a little bit, and I have to admit I do not understand what it does... :(

If some Ironic falks can explain, that'd be great!

Cheers,

Thomas Goirand (zigo)

On 12/3/21 10:41 PM, Julia Kreger wrote:

Extreme magic - OpenStack Edition. At least that feels like the answer.

In essence, we create VMs. We attach them to the network, then configure virtualbmc or sushy-tools to play the role of a BMC and one of the things that library understand is the giant switch of "boot to network". From there we ipxe boot. The VMs play the role of baremetal machines through the emulation of the BMC and they are just hooked up to the network.

This does require the job to use root privileges to setup networking/vms. And the virtualbmc or sushy-tools service needs enough rights to toggle the VM's config. Would be happy to try and discuss specifics of the devstack plugin, but I'm unsure if that would really help. Regardless, you know where to find us. :)

As an aside, when I first read the post I was thinking of booting from ISO images. I've known people to embed ipxe into disk images, and you can embed static config/instructions for booting into an ipxe binary. It does require custom binary builds, but it might make what you're trying to achieve easier.

Hi Julia!

Thanks for your answer.

Well, I've read multiple times your reply, but I have to admit I still don't get all the details. For example, how would you "embed ipxe into disk images" for example? Should I for example run the iPXE binary from something like syslinux (or grub?)? But isn't iPXE usually running *after* the BIOS of a PC gets an IP from DHCP?

I've just read a post where someone boots the iPXE binary from Grub, so that part shouldn't be hard: format a disk, make a ext4 partition, put the iPXE binary there, and tell grub to execute it. Is that all?!?

Cheers,

Thomas Goirand (zigo)

P.S: I do have some knowledge on how to prepare disk images. After all, I've maintained the Debian OpenStack images for 3 Debian releases. So I probably just need a few pointers...

On 12/3/21 11:34 PM, Alex Schultz wrote:

On Fri, Dec 3, 2021 at 3:22 PM Thomas Goirand zigo@debian.org wrote:

...

On 12/3/21 10:41 PM, Julia Kreger wrote:

...

Extreme magic - OpenStack Edition. At least that feels like the answer.

In essence, we create VMs. We attach them to the network, then configure virtualbmc or sushy-tools to play the role of a BMC and one of the things that library understand is the giant switch of "boot to network". From there we ipxe boot. The VMs play the role of baremetal machines through the emulation of the BMC and they are just hooked up to the network.

This does require the job to use root privileges to setup networking/vms. And the virtualbmc or sushy-tools service needs enough rights to toggle the VM's config. Would be happy to try and discuss specifics of the devstack plugin, but I'm unsure if that would really help. Regardless, you know where to find us. :)

As an aside, when I first read the post I was thinking of booting from ISO images. I've known people to embed ipxe into disk images, and you can embed static config/instructions for booting into an ipxe binary. It does require custom binary builds, but it might make what you're trying to achieve easier.

Hi Julia!

Thanks for your answer.

Well, I've read multiple times your reply, but I have to admit I still don't get all the details. For example, how would you "embed ipxe into disk images" for example? Should I for example run the iPXE binary from something like syslinux (or grub?)? But isn't iPXE usually running *after* the BIOS of a PC gets an IP from DHCP?

I'm sure there are other examples around but the one we used to (maybe still do) use for ovb can be built via:

https://opendev.org/openstack/openstack-virtual-baremetal/src/branch/master/...

You basically build an image that just attempts ipxe on repeat.

Hi Alex!

In fact, I just found out my solution: http://boot.ipxe.org/ipxe.iso

Just download this, then:

openstack image create \ --disk-format iso \ --container-format bare \ --file ipxe.iso ipxe.iso

Et voila! The VM boots from PXE (I checked on the VNC console, and it also gets DHCP). I didn't go further, but it should be easy to setup the rest of the components (ie: a private network without DHCP, and a VM acting as DHCPd + tftp-hpa).

I'll let you guys know how much progress I'm making with all of this (probably I'll start working on this next week...).

Cheers,

Thomas Goirand (zigo)

On Fri, 2021-12-03 at 15:54 -0700, Alex Schultz wrote:

On Fri, Dec 3, 2021 at 3:45 PM Thomas Goirand zigo@debian.org wrote:

...

On 12/3/21 11:34 PM, Alex Schultz wrote:

...

On Fri, Dec 3, 2021 at 3:22 PM Thomas Goirand zigo@debian.org wrote:

...

On 12/3/21 10:41 PM, Julia Kreger wrote:

...

Extreme magic - OpenStack Edition. At least that feels like the answer.

In essence, we create VMs. We attach them to the network, then configure virtualbmc or sushy-tools to play the role of a BMC and one of the things that library understand is the giant switch of "boot to network". From there we ipxe boot. The VMs play the role of baremetal machines through the emulation of the BMC and they are just hooked up to the network.

This does require the job to use root privileges to setup networking/vms. And the virtualbmc or sushy-tools service needs enough rights to toggle the VM's config. Would be happy to try and discuss specifics of the devstack plugin, but I'm unsure if that would really help. Regardless, you know where to find us. :)

As an aside, when I first read the post I was thinking of booting from ISO images. I've known people to embed ipxe into disk images, and you can embed static config/instructions for booting into an ipxe binary. It does require custom binary builds, but it might make what you're trying to achieve easier.

Hi Julia!

Thanks for your answer.

Well, I've read multiple times your reply, but I have to admit I still don't get all the details. For example, how would you "embed ipxe into disk images" for example? Should I for example run the iPXE binary from something like syslinux (or grub?)? But isn't iPXE usually running *after* the BIOS of a PC gets an IP from DHCP?

I'm sure there are other examples around but the one we used to (maybe still do) use for ovb can be built via:

https://opendev.org/openstack/openstack-virtual-baremetal/src/branch/master/...

You basically build an image that just attempts ipxe on repeat.

Hi Alex!

In fact, I just found out my solution: http://boot.ipxe.org/ipxe.iso

Yea I think that's what gets built with make and then it qemu-img converts it for you.

...

Just download this, then:

openstack image create \ --disk-format iso \ --container-format bare \ --file ipxe.iso ipxe.iso

Et voila! The VM boots from PXE (I checked on the VNC console, and it also gets DHCP). I didn't go further, but it should be easy to setup the rest of the components (ie: a private network without DHCP, and a VM acting as DHCPd + tftp-hpa).

Yea as previously mentioned you may just need to disable port security and it should be fine. This is basically what we (tripleo) do in our 3rd party testing where the dhcpd is from the undercloud. I've also used the same image locally in libvirt + sushy-emulator for testing as well.

disabling port security might help but in general pxe booting nova instance is not supported. as in this is intentally not a supproted usecase currently, not that it coudl not be enable but we dont intend it to work outside of a ironic context. disbaling port security really shoudl only be required if you want to run your own dhcp server, if you willing ot manually pxe boot interactivly you have ohter options.

we also dont expclitly block pxe booting if the qemu seabios/uefi firmware support it so you can enable the boot menu vai hw:boot_menu=True extra spec or hw_boot_menu=True image property https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt... and select pxe boot if the firmware on the host loaded by qemu support it, but nova makes no guarentee that it will work with the libvirt driver.

ironic is sort of a special case with regards to allowing/using pxe booting when its operating as a nova virt driver. in ironics case it more that nova does not care how the ironic servcei interally boots the instance and ironci just happens to use ipmi and pxe/ipxe as one of the options. you do not nesssiasarly need to use an ipxe boot image to pxeboot either as the seabios image, depening on your host useually has pxe supprot built in. your milage will vary however as this is outside the bounds of what is intended to work so if you encounterd issues adressing those would be new feature blueprint not bugs so they would not be backportable.

if your intent is to just use this for testing then following in the path of tripleo and ironic by using virtualbmc to provide ipmi access to the vm might be ok but that assuems you are deploying an openstack cloud which you can then ssh into the hosts to install an run the virtual bmc.

if you really want to do this in an unaltered cloud we would nee to configre the vm to boot form the network https://libvirt.org/formatdomain.html#bios-bootloader https://libvirt.org/formatdomain.html#specifying-boot-order

im not sure what the best way to extend nova to support that would be, likely a neutron extention to mark ports a bootable which nova would read and then use to update the xml. host config is not really an option as it would be problematic for move operations, but flavor extra specs or image properites would be viable alternitive to the neutron approch. although neutorn ports would give a better ux. if you dont make progress with other approchs then you should consider fileing a spec for network boot support in nova/neutron.

...

I'll let you guys know how much progress I'm making with all of this (probably I'll start working on this next week...).

Cheers,

Thomas Goirand (zigo)

On 03/12/2021 21:01, Cedric Lemarchand wrote:

We faced the same use case, and we end up doing something like this:

• disable security on the network provider (or on the instance port

where pxe boot is needed)

• deploy or rescue the intance with an ipxe image

As for DHCP, you can very much use the OpenStack DHCP and just set the options like next-server / server-ip-address

* https://docs.openstack.org/api-ref/network/v2/?expanded=update-port-detail#e...  * https://docs.openstack.org/neutron/latest/ovn/dhcp_opts.html

and this IP then points to your VM with the TFTP and other PXE / iPXE tooling running.

Regards

Christian

So I validated what from my point of view seems to be the most "supported" way to boot an Openstack instance in PXE :

- add the flavor the extra specs "hw:boot_menu = 1" (thanks Sean) - spawn the instance - configure the extra_dhcp_opts of the associated port (thanks Christian). Its not supported with the CLI, but doable with an sdk, below a Pythonic example [1] - reboot the instance, hit esc at the bios prompt then third choice (iPXE)

This allows the use of the Openstack DHCP and to keep port security enabled, which is nice.

Cheers

[1] --- conn.update_port(port_id, extra_dhcp_opts=[ {"opt_name": "server-ip-address","opt_value": "10.0.0.15"}, {"opt_name": "bootfile-name", "opt_value": "pxelinux.0"}, {"opt_name": "tftp-server", "opt_value": "10.0.0.15"} ] ) ---

On Wed, Dec 8, 2021 at 8:52 PM Cedric Lemarchand yipikai7@gmail.com wrote:

Thanks for the heads up.

Pointer is about ovn, I didnt find anything regarding ovs.

Did the Neutron API support these options both for ovn and ovs ?

On Wed, Dec 8, 2021, 17:12 Christian Rohmann christian.rohmann@inovex.de wrote:

...

On 03/12/2021 21:01, Cedric Lemarchand wrote:

...

We faced the same use case, and we end up doing something like this:

• disable security on the network provider (or on the instance port

where pxe boot is needed)

• deploy or rescue the intance with an ipxe image

As for DHCP, you can very much use the OpenStack DHCP and just set the options like next-server / server-ip-address

https://docs.openstack.org/api-ref/network/v2/?expanded=update-port-detail#e...

• https://docs.openstack.org/neutron/latest/ovn/dhcp_opts.html

and this IP then points to your VM with the TFTP and other PXE / iPXE tooling running.

Regards

Christian

On Tue, Dec 14, 2021 at 12:10 PM Thomas Goirand zigo@debian.org wrote:

It really is supported by the cli. I've done this way:

Right, thing is I was trying to update the port configuration rather than creating it before the instance, good to know thanks.