Fwd: [ironic] IPA image does not want to boot with UEFI
Hi Julia, thanks for the response.
I have now verified that the server has Secure Boot set to Disabled. It happens on all our HPE Gen10 and Gen10+ servers that are booting with UEFI in the same configuration (Secure Boot Disabled).
Unsigned initrd from ubuntu for example works with Kernel from IPA:
I have also updated firmware of the server to the latest version, but it is still the same.
I am going to try to drill deeper in IPA builder code and IPA initrd which seems not to go well with UEFI boot mode.
I am grateful for any hint.
Bests, Vuk
Hi! Out of curiosity, how did you generate the esp? Normally Ironic will build a grub.cfg file and include it into the iso, then it is loaded automatically (providing the grub_config_path is properly set) so you don't have anything to type by hand... That grub.cfg default template uses "linuxefi" and "initrdefi" (instead of linux/initrd you are using) which seem correct for most distributions (but depending on the way you built the esp, if you used grub from sources for example, linux/initrd would still be the right commands). Maybe you can try the "efi" variants.
Best of luck! Francois (frigo)
On Tue, 11 May 2021 at 20:45, Vuk Gojnic vuk.gojnic@gmail.com wrote:
Hi Julia, thanks for the response.
I have now verified that the server has Secure Boot set to Disabled. It happens on all our HPE Gen10 and Gen10+ servers that are booting with UEFI in the same configuration (Secure Boot Disabled).
Unsigned initrd from ubuntu for example works with Kernel from IPA:
I have also updated firmware of the server to the latest version, but it is still the same.
I am going to try to drill deeper in IPA builder code and IPA initrd which seems not to go well with UEFI boot mode.
I am grateful for any hint.
Bests, Vuk
Hi Francois,
Thanks for the reply.
I am using esp from Ubuntu install iso. Intentionally go to Grub prompt to play with kernel parameters in attempt to debug.
linuxefi/initrdefi had seemingly same behaviour like linux/initrd.
What I actually just detected is that if I try to do “ls” of the device where kernel and initrd are residing it lists it correctly, but if I do “ls -la” it lists couple of files, but when it comes to initrd it freezes and I get again “Red Screen of Death” with same errors.
Initrd that works just fine is 52MB big and uncompressed, while the one That blocks is 406MB big compressed. I started suspecting that the problem us is filesize limit.
-Vuk
Sent from my Telekom.de iPhone 14 prototype
On 11. May 2021, at 21:28, Francois rigault.francois@gmail.com wrote:
Hi! Out of curiosity, how did you generate the esp? Normally Ironic will build a grub.cfg file and include it into the iso, then it is loaded automatically (providing the grub_config_path is properly set) so you don't have anything to type by hand... That grub.cfg default template uses "linuxefi" and "initrdefi" (instead of linux/initrd you are using) which seem correct for most distributions (but depending on the way you built the esp, if you used grub from sources for example, linux/initrd would still be the right commands). Maybe you can try the "efi" variants.
Best of luck! Francois (frigo)
On Tue, 11 May 2021 at 20:45, Vuk Gojnic vuk.gojnic@gmail.com wrote:
Hi Julia, thanks for the response.
I have now verified that the server has Secure Boot set to Disabled. It happens on all our HPE Gen10 and Gen10+ servers that are booting with UEFI in the same configuration (Secure Boot Disabled).
Unsigned initrd from ubuntu for example works with Kernel from IPA:
I have also updated firmware of the server to the latest version, but it is still the same.
I am going to try to drill deeper in IPA builder code and IPA initrd which seems not to go well with UEFI boot mode.
I am grateful for any hint.
Bests, Vuk
Hello everybody,
I have finally found the root cause of the problem. It is indeed in the size of "initramfs" that ironic-python-agent-builder produces. They are all normally 350-420 MB large (compressed). However, since boot protocol 2.03 kernel explicitly limits highest initrd address available to the bootloader (https://www.kernel.org/doc/Documentation/x86/boot.txt), thus limits the size of initrd that could be used. Therefore none of production IPA initramfs images from official location: https://tarballs.openstack.org/ironic-python-agent/dib/files/, nor custom made by ironic-python-agent-builder can not boot. The bootloader is just crashing on them.
When I turned to TinyIPA from here: https://tarballs.openstack.org/ironic-python-agent/tinyipa/files/ (which are explicitly marked not for production), it worked well and booted the IPA. So I will proceed with Tiny, but it might be useful to give hint of to others.
Thanks anyway for support from everybody here.
See you!
-Vuk
They are in the range of 400MB
On Tue, May 11, 2021 at 11:08 PM Vuk Gojnic vuk.gojnic@gmail.com wrote:
Hi Francois,
Thanks for the reply.
I am using esp from Ubuntu install iso. Intentionally go to Grub prompt to play with kernel parameters in attempt to debug.
linuxefi/initrdefi had seemingly same behaviour like linux/initrd.
What I actually just detected is that if I try to do “ls” of the device where kernel and initrd are residing it lists it correctly, but if I do “ls -la” it lists couple of files, but when it comes to initrd it freezes and I get again “Red Screen of Death” with same errors.
Initrd that works just fine is 52MB big and uncompressed, while the one That blocks is 406MB big compressed. I started suspecting that the problem us is filesize limit.
-Vuk
Sent from my Telekom.de iPhone 14 prototype
On 11. May 2021, at 21:28, Francois rigault.francois@gmail.com wrote:
Hi! Out of curiosity, how did you generate the esp? Normally Ironic will build a grub.cfg file and include it into the iso, then it is loaded automatically (providing the grub_config_path is properly set) so you don't have anything to type by hand... That grub.cfg default template uses "linuxefi" and "initrdefi" (instead of linux/initrd you are using) which seem correct for most distributions (but depending on the way you built the esp, if you used grub from sources for example, linux/initrd would still be the right commands). Maybe you can try the "efi" variants.
Best of luck! Francois (frigo)
I'm glad that it worked for you!
Before others follow your advice: the difference in size in DIB builds and tinyIPA is mostly because of firmware and kernel modules. If tinyIPA does not work for you or behaves in a weird way (no disks detected, some NICs not detected), then you're stuck with DIB builds.
Vuk, there is one more option you could exercise. IPA-builder supports an --lzma flag to pack the initramfs with a more efficient algorithm: https://opendev.org/openstack/ironic-python-agent-builder/src/branch/master/... .
Dmitry
On Wed, May 12, 2021 at 2:33 PM Vuk Gojnic vuk.gojnic@gmail.com wrote:
Hello everybody,
I have finally found the root cause of the problem. It is indeed in the size of "initramfs" that ironic-python-agent-builder produces. They are all normally 350-420 MB large (compressed). However, since boot protocol 2.03 kernel explicitly limits highest initrd address available to the bootloader (https://www.kernel.org/doc/Documentation/x86/boot.txt), thus limits the size of initrd that could be used. Therefore none of production IPA initramfs images from official location: https://tarballs.openstack.org/ironic-python-agent/dib/files/, nor custom made by ironic-python-agent-builder can not boot. The bootloader is just crashing on them.
When I turned to TinyIPA from here: https://tarballs.openstack.org/ironic-python-agent/tinyipa/files/ (which are explicitly marked not for production), it worked well and booted the IPA. So I will proceed with Tiny, but it might be useful to give hint of to others.
Thanks anyway for support from everybody here.
See you!
-Vuk
They are in the range of 400MB
On Tue, May 11, 2021 at 11:08 PM Vuk Gojnic vuk.gojnic@gmail.com wrote:
Hi Francois,
Thanks for the reply.
I am using esp from Ubuntu install iso. Intentionally go to Grub prompt
to play with kernel parameters in attempt to debug.
linuxefi/initrdefi had seemingly same behaviour like linux/initrd.
What I actually just detected is that if I try to do “ls” of the device
where kernel and initrd are residing it lists it correctly, but if I do “ls -la” it lists couple of files, but when it comes to initrd it freezes and I get again “Red Screen of Death” with same errors.
Initrd that works just fine is 52MB big and uncompressed, while the one
That blocks is 406MB big compressed. I started suspecting that the problem us is filesize limit.
-Vuk
Sent from my Telekom.de iPhone 14 prototype
On 11. May 2021, at 21:28, Francois rigault.francois@gmail.com
wrote:
Hi! Out of curiosity, how did you generate the esp? Normally Ironic will build a grub.cfg file and include it into the iso, then it is loaded automatically (providing the grub_config_path is properly set) so you don't have anything to type by hand... That grub.cfg default template uses "linuxefi" and "initrdefi" (instead of linux/initrd you are using) which seem correct for most distributions (but depending on the way you built the esp, if you used grub from sources for example, linux/initrd would still be the right commands). Maybe you can try the "efi" variants.
Best of luck! Francois (frigo)
Hi Dmitry,
Thanks for additional tipps. When investigating the initrd I have noticed that the most of the space goes on firmware and modules/drivers. If we notice something not working with TinyIPA we can probably cherry-pick the modules that we need and leave everything else out and that way get smaller image.
I have another question though - do you know how could we make Kernel/Grub accept to boot large initrd? How are other folks doing it? I assume not everybody is just using TinyIPA for production...
Tnx!
Vuk
On Wed, May 12, 2021 at 5:36 PM Dmitry Tantsur dtantsur@redhat.com wrote:
I'm glad that it worked for you!
Before others follow your advice: the difference in size in DIB builds and tinyIPA is mostly because of firmware and kernel modules. If tinyIPA does not work for you or behaves in a weird way (no disks detected, some NICs not detected), then you're stuck with DIB builds.
Vuk, there is one more option you could exercise. IPA-builder supports an --lzma flag to pack the initramfs with a more efficient algorithm: https://opendev.org/openstack/ironic-python-agent-builder/src/branch/master/....
Dmitry
Hi,
I'm not sure. We have never hit this problem with DIB-built images before. I know that TripleO uses an even larger image than one we publish on tarballs.o.o.
Dmitry
On Wed, May 12, 2021 at 8:10 PM Vuk Gojnic vuk.gojnic@gmail.com wrote:
Hi Dmitry,
Thanks for additional tipps. When investigating the initrd I have noticed that the most of the space goes on firmware and modules/drivers. If we notice something not working with TinyIPA we can probably cherry-pick the modules that we need and leave everything else out and that way get smaller image.
I have another question though - do you know how could we make Kernel/Grub accept to boot large initrd? How are other folks doing it? I assume not everybody is just using TinyIPA for production...
Tnx!
Vuk
On Wed, May 12, 2021 at 5:36 PM Dmitry Tantsur dtantsur@redhat.com wrote:
I'm glad that it worked for you!
Before others follow your advice: the difference in size in DIB builds
and tinyIPA is mostly because of firmware and kernel modules. If tinyIPA does not work for you or behaves in a weird way (no disks detected, some NICs not detected), then you're stuck with DIB builds.
Vuk, there is one more option you could exercise. IPA-builder supports
an --lzma flag to pack the initramfs with a more efficient algorithm: https://opendev.org/openstack/ironic-python-agent-builder/src/branch/master/... .
Dmitry
I found where was my issue. After using different GRUBX64.efi and BOOTX64.efi binaries (this time from https://vault.centos.org/8.3.2011/BaseOS/x86_64/kickstart/EFI/BOOT/ instead from Ubuntu Bionic LiveCD) everything worked normally and the large initrd was successfully loaded. It seems that used EFI binaries from Ubuntu had that issue.
Advice in such case is: check with another bootloader variant/version if the problem persists.
Thanks!
-Vuk
On Mon, May 17, 2021 at 4:14 PM Dmitry Tantsur dtantsur@redhat.com wrote:
Hi,
I'm not sure. We have never hit this problem with DIB-built images before. I know that TripleO uses an even larger image than one we publish on tarballs.o.o.
participants (3)
-
Dmitry Tantsur
-
Francois
-
Vuk Gojnic