On Wed, Oct 7, 2020 at 11:16 AM Mark Goddard <mark@stackhpc.com> wrote:

On Wed, 7 Oct 2020 at 16:11, Riccardo Pittau <elfosardo@gmail.com> wrote:

...

Hello fellow openstackers!

At the moment it's not possible to migrate the ironic-python-agent-builder src jobs from bionic to focal nodeset because of diskimage-builder limitations. We're stuck with ubuntu bionic and we're pinning those jobs to the bionic nodeset for the time being: https://review.opendev.org/756291

One of the community goals for victoria is to move the base nodeset of the CI jobs from ubuntu bionic to focal. In general, doing this for most of the ironic projects has not been trivial, but still doable, and it has been accomplished almost entirely. The biggest challenge comes from the src jobs in ironic-python-agent-builder where, for some of them, we build ironic-python-agent ramdisks using rpm-based distributions (mainly centos) with diskimage-builder on ubuntu bionic. This is possible using utilities (e.g. yumdownloader) included in packages still present in the ubuntu repositories, such as yum-utils and rpm. Starting from Ubuntu focal, the yum-utils package has been removed from the repositories because of lack of support of Python 2.x and there's no plan to provide such support, at least to my knowledge. The alternative provided by dnf is not usable as there's also no plan to compile and provide a package of dnf for deb-based distributions. For the reasons mentioned above, currently the ironic project team can't complete the migration of the CI jobs from bionic to focal and there's no ETA on when this can be accomplished.

Considering all the things in the preamble, two possibilities are available: - change the mechanics in diskimage-builder; this process would completely change the way DIB builds rpm-based distros; this approach delegates the work almost entirely to the DIB team. - instead of migrating to focal, migrate to centos-8 nodeset; that would mean having devstack+ironic working on centos-8, which poses an interesting challenge and would consume no little resources from the ironic team.

Opinions and advice are very welcome!

My first reaction would be to consider the user impact beyond the box checking of fulfilling a goal. Does this imply Ubuntu users can no longer build IPA images from Focal? That would be a shame.

The goals are really meant to drive the community together as a group. And if box checking is not appropriate, then it is not appropriate. I think the key is in the meaning of the goal which is to drive everyone forward together. I do concur end user impact is the key item to focus on. I don't think this is helped due to pre-existing constraints. I seem to remember that you couldn't build ubuntu on centos previously, so this sort of issue does not surprise me. At least, not without having some extra packages present that one could install and it might work with some hope. My guess is that we're effectively entering a situation where if I want to build a centos/rhel/fedora IPA image, I need to run the ipa builder command on one of those machine types, and if I want the same for debian/ubuntu, I need to run the build on one of those operating systems. Is that situation horrible for users, not really because they are and likely should keep the distribution the same for familiarity and compatibility. The thing we likely need to do is do an ubuntu IPA image test in CI but not save the artifact. Or debian!

In terms of fulfilling the goal, perhaps running DIB in a CentOS container would help? This of course adds complexity, potential problems, and doesn't really test what users would do.

...

Thanks,

Riccardo

On Wed, Oct 7, 2020 at 1:51 PM Mark Goddard <mark@stackhpc.com> wrote:

On Wed, 7 Oct 2020, 21:18 Julia Kreger, <juliaashleykreger@gmail.com> wrote:

...

On Wed, Oct 7, 2020 at 11:16 AM Mark Goddard <mark@stackhpc.com> wrote:

...

On Wed, 7 Oct 2020 at 16:11, Riccardo Pittau <elfosardo@gmail.com> wrote:

...

Hello fellow openstackers!

At the moment it's not possible to migrate the ironic-python-agent-builder src jobs from bionic to focal nodeset because of diskimage-builder limitations. We're stuck with ubuntu bionic and we're pinning those jobs to the bionic nodeset for the time being: https://review.opendev.org/756291

One of the community goals for victoria is to move the base nodeset of the CI jobs from ubuntu bionic to focal. In general, doing this for most of the ironic projects has not been trivial, but still doable, and it has been accomplished almost entirely. The biggest challenge comes from the src jobs in ironic-python-agent-builder where, for some of them, we build ironic-python-agent ramdisks using rpm-based distributions (mainly centos) with diskimage-builder on ubuntu bionic. This is possible using utilities (e.g. yumdownloader) included in packages still present in the ubuntu repositories, such as yum-utils and rpm. Starting from Ubuntu focal, the yum-utils package has been removed from the repositories because of lack of support of Python 2.x and there's no plan to provide such support, at least to my knowledge. The alternative provided by dnf is not usable as there's also no plan to compile and provide a package of dnf for deb-based distributions. For the reasons mentioned above, currently the ironic project team can't complete the migration of the CI jobs from bionic to focal and there's no ETA on when this can be accomplished.

Considering all the things in the preamble, two possibilities are available: - change the mechanics in diskimage-builder; this process would completely change the way DIB builds rpm-based distros; this approach delegates the work almost entirely to the DIB team. - instead of migrating to focal, migrate to centos-8 nodeset; that would mean having devstack+ironic working on centos-8, which poses an interesting challenge and would consume no little resources from the ironic team.

Opinions and advice are very welcome!

My first reaction would be to consider the user impact beyond the box checking of fulfilling a goal. Does this imply Ubuntu users can no longer build IPA images from Focal? That would be a shame.

The goals are really meant to drive the community together as a group. And if box checking is not appropriate, then it is not appropriate. I think the key is in the meaning of the goal which is to drive everyone forward together. I do concur end user impact is the key item to focus on. I don't think this is helped due to pre-existing constraints. I seem to remember that you couldn't build ubuntu on centos previously, so this sort of issue does not surprise me. At least, not without having some extra packages present that one could install and it might work with some hope.

I didn't mean to imply that the goal is not a worthwhile endeavour, only that user impact should come first.

...

I never thought you were trying to imply that for a second! But your raising a great point and I guess I'm also kind of thinking maybe the goal doesn't always fit even though it is worthwhile.

...

My guess is that we're effectively entering a situation where if I want to build a centos/rhel/fedora IPA image, I need to run the ipa builder command on one of those machine types, and if I want the same for debian/ubuntu, I need to run the build on one of those operating systems. Is that situation horrible for users, not really because they are and likely should keep the distribution the same for familiarity and compatibility. The thing we likely need to do is do an ubuntu IPA image test in CI but not save the artifact. Or debian!

This does raise the question of how to test a centos based IPA image before it is published though.

The thought that comes to mind is that we could hybridize and run some jobs on centos some on ubuntu. At least that is my thought. I see Clark has sent a reply, so I'm off to read that! :)

...

...

In terms of fulfilling the goal, perhaps running DIB in a CentOS container would help? This of course adds complexity, potential problems, and doesn't really test what users would do.

...

Thanks,

Riccardo

On Wed, Oct 7, 2020, at 1:18 PM, Julia Kreger wrote:

On Wed, Oct 7, 2020 at 11:16 AM Mark Goddard <mark@stackhpc.com> wrote:

...

On Wed, 7 Oct 2020 at 16:11, Riccardo Pittau <elfosardo@gmail.com> wrote:

...

Hello fellow openstackers!

At the moment it's not possible to migrate the ironic-python-agent-builder src jobs from bionic to focal nodeset because of diskimage-builder limitations. We're stuck with ubuntu bionic and we're pinning those jobs to the bionic nodeset for the time being: https://review.opendev.org/756291

One of the community goals for victoria is to move the base nodeset of the CI jobs from ubuntu bionic to focal. In general, doing this for most of the ironic projects has not been trivial, but still doable, and it has been accomplished almost entirely. The biggest challenge comes from the src jobs in ironic-python-agent-builder where, for some of them, we build ironic-python-agent ramdisks using rpm-based distributions (mainly centos) with diskimage-builder on ubuntu bionic. This is possible using utilities (e.g. yumdownloader) included in packages still present in the ubuntu repositories, such as yum-utils and rpm. Starting from Ubuntu focal, the yum-utils package has been removed from the repositories because of lack of support of Python 2.x and there's no plan to provide such support, at least to my knowledge. The alternative provided by dnf is not usable as there's also no plan to compile and provide a package of dnf for deb-based distributions. For the reasons mentioned above, currently the ironic project team can't complete the migration of the CI jobs from bionic to focal and there's no ETA on when this can be accomplished.

Considering all the things in the preamble, two possibilities are available: - change the mechanics in diskimage-builder; this process would completely change the way DIB builds rpm-based distros; this approach delegates the work almost entirely to the DIB team. - instead of migrating to focal, migrate to centos-8 nodeset; that would mean having devstack+ironic working on centos-8, which poses an interesting challenge and would consume no little resources from the ironic team.

Opinions and advice are very welcome!

My first reaction would be to consider the user impact beyond the box checking of fulfilling a goal. Does this imply Ubuntu users can no longer build IPA images from Focal? That would be a shame.

The goals are really meant to drive the community together as a group. And if box checking is not appropriate, then it is not appropriate. I think the key is in the meaning of the goal which is to drive everyone forward together. I do concur end user impact is the key item to focus on. I don't think this is helped due to pre-existing constraints. I seem to remember that you couldn't build ubuntu on centos previously, so this sort of issue does not surprise me. At least, not without having some extra packages present that one could install and it might work with some hope.

My guess is that we're effectively entering a situation where if I want to build a centos/rhel/fedora IPA image, I need to run the ipa builder command on one of those machine types, and if I want the same for debian/ubuntu, I need to run the build on one of those operating systems. Is that situation horrible for users, not really because they are and likely should keep the distribution the same for familiarity and compatibility. The thing we likely need to do is do an ubuntu IPA image test in CI but not save the artifact. Or debian!

There has actually been some thought on this problem recently. Those tools are all used to bootstrap a chroot with the necessary components to then build the rest of the image. Fortunately there are other approaches that can be taken in DIB to get to that point. Some of the elements start with a preexisting cloud image for example (ubuntu does this where ubuntu-minimal starts with debootstrap). More recently the thought has been that we should probably think about bootstrapping with docker images because basically all the distros publish such a thing. That element exists and is called "docker" [0] but may need more testing [1]. This is attractive because it means we should be able to run on just about any distro as long as the DIB host's kernel doesn't conflict with the userspace in the docker container used to bootstrap image building. I think the work here has largely stalled out, but I'm sure help would be welcome if Ironic and others think this is a useful approach to take. I'm not in a great spot to pick this up myself, but can probably help out if someone else is driving it (reviews, testing assistance, etc) [0] https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_... [1] https://review.opendev.org/#/c/700041/

On Thu, 8 Oct 2020 at 05:20, Ian Wienand <iwienand@redhat.com> wrote:

On Wed, Oct 07, 2020 at 05:09:56PM +0200, Riccardo Pittau wrote:

...

This is possible using utilities (e.g. yumdownloader) included in packages still present in the ubuntu repositories, such as yum-utils and rpm. Starting from Ubuntu focal, the yum-utils package has been removed from the repositories because of lack of support of Python 2.x and there's no plan to provide such support, at least to my knowledge.

Yes, this is a problem for the "-minimal" elements that build an non-native chroot environment. Similar issues have occured with Suse and the zypper package manager not being available on the build host.

The options I can see:

- use the native build-host; i.e. build on centos as you described

- the non-minimal, i.e. "centos" and "suse", for example, images might work under the current circumstances. They use the upsream ISO to create the initial chroot. These are generally bigger, and we've had stability issues in the past with the upstream images changing suddenly in various ways that were a maintenance headache.

- use a container for dib. DIB doesn't have a specific container, but is part of the nodepool-builder container [1]. This is ultimately based on Debian buster [2] which has enough support to build everything ... for now. As noted this doesn't really solve the problem indefinitely, but certainly buys some time if you run dib out of that container (we could, of course, make a separate dib container; but it would be basically the same just without nodepool in it). This is what OpenDev production is using now, and all the CI is ultimately based on this container environment.

If this could be wrapped up in a DIB-like command, this seems the most flexible to me.

- As clarkb has mentioned, probably the most promising alternative is to use the upstream container images as the basis for the initial chroot environments. jeblair has done most of this work with [3]. I'm fiddling with it to merge to master and see what's up ... I feel like maybe there were bootloader issues, although the basic extraction was working. This will allow the effort put into existing elements to not be lost.

Initial reaction is that this would suffer from the same problems as using a cloud image as the base, but worse. Container images are seen as disposable, and who knows what measures might have been taken to reduce their size and disable/remove the init system?

If I had to pick; I'd probably say that using the nodepool-builder container is the best path. That has the most momentum behind it because it's used for the OpenDev image builds. As we work on the container-image base elements, this work will be deployed into the container (meaning the container is less reliant on the underlying version of Debian) and you can switch to them as appropriate.

-i

[1] https://hub.docker.com/r/zuul/nodepool-builder [2] https://opendev.org/opendev/system-config/src/branch/master/docker/python-ba... [3] https://review.opendev.org/#/c/700083/

On Thu, Oct 8, 2020 at 10:19 AM Luigi Toscano <ltoscano@redhat.com> wrote:

...

On Wed, Oct 07, 2020 at 05:09:56PM +0200, Riccardo Pittau wrote:

...

This is possible using utilities (e.g. yumdownloader) included in

...

...

still present in the ubuntu repositories, such as yum-utils and rpm. Starting from Ubuntu focal, the yum-utils package has been removed from the repositories because of lack of support of Python 2.x and there's no

On Thursday, 8 October 2020 06:18:35 CEST Ian Wienand wrote: packages plan

...

...

to provide such support, at least to my knowledge.

Yes, this is a problem for the "-minimal" elements that build an non-native chroot environment. Similar issues have occured with Suse and the zypper package manager not being available on the build host.

The options I can see:

- use the native build-host; i.e. build on centos as you described

- the non-minimal, i.e. "centos" and "suse", for example, images might work under the current circumstances. They use the upsream ISO to create the initial chroot. These are generally bigger, and we've had stability issues in the past with the upstream images changing suddenly in various ways that were a maintenance headache.

- use a container for dib. DIB doesn't have a specific container, but is part of the nodepool-builder container [1]. This is ultimately based on Debian buster [2] which has enough support to build everything ... for now. As noted this doesn't really solve the problem indefinitely, but certainly buys some time if you run dib out of that container (we could, of course, make a separate dib container; but it would be basically the same just without nodepool in it). This is what OpenDev production is using now, and all the CI is ultimately based on this container environment.

- As clarkb has mentioned, probably the most promising alternative is to use the upstream container images as the basis for the initial chroot environments. jeblair has done most of this work with [3]. I'm fiddling with it to merge to master and see what's up ... I feel like maybe there were bootloader issues, although the basic extraction was working. This will allow the effort put into existing elements to not be lost.

If I had to pick; I'd probably say that using the nodepool-builder container is the best path. That has the most momentum behind it because it's used for the OpenDev image builds. As we work on the container-image base elements, this work will be deployed into the container (meaning the container is less reliant on the underlying version of Debian) and you can switch to them as appropriate.

I have to mention at this point, at risk of reharshing old debates, that an alternative in various scenarios (maybe not all) is the usage of libguestfs and its tools which modifies an existing base image.

https://libguestfs.org/

We switched to it in Sahara for most of the guest images and that saved some headaches when building from a different host. https://docs.openstack.org/sahara/latest/user/building-guest-images.html

I like guestfish (a lot), but our IPA images are a bit awkward: we take a qcow2 image and convert it to a kernel/ramdisk pair. I guess we can use guestfish to get the former, but not the latter. It also means changing the approach that has been used and documented for years. Not impossible, but should not be done lightly. Dmitry

I'd like to mention that libguestfs has been carrying a virt-dib tool for a while, but it has been tested only back to a certain version of dib: https://libguestfs.org/virt-dib.1.html

-- Luigi

-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

On Thursday, 8 October 2020 17:38:57 CEST Clark Boylan wrote:

On Thu, Oct 8, 2020, at 1:17 AM, Luigi Toscano wrote:

...

On Thursday, 8 October 2020 06:18:35 CEST Ian Wienand wrote:

...

On Wed, Oct 07, 2020 at 05:09:56PM +0200, Riccardo Pittau wrote:

...

This is possible using utilities (e.g. yumdownloader) included in packages still present in the ubuntu repositories, such as yum-utils and rpm. Starting from Ubuntu focal, the yum-utils package has been removed from the repositories because of lack of support of Python 2.x and there's no plan to provide such support, at least to my knowledge.

Yes, this is a problem for the "-minimal" elements that build an non-native chroot environment. Similar issues have occured with Suse and the zypper package manager not being available on the build host.

The options I can see:

- use the native build-host; i.e. build on centos as you described

- the non-minimal, i.e. "centos" and "suse", for example, images might

work under the current circumstances. They use the upsream ISO to create the initial chroot. These are generally bigger, and we've had stability issues in the past with the upstream images changing suddenly in various ways that were a maintenance headache.

- use a container for dib. DIB doesn't have a specific container, but

is part of the nodepool-builder container [1]. This is ultimately based on Debian buster [2] which has enough support to build everything ... for now. As noted this doesn't really solve the problem indefinitely, but certainly buys some time if you run dib out of that container (we could, of course, make a separate dib container; but it would be basically the same just without nodepool in it). This is what OpenDev production is using now, and all the CI is ultimately based on this container environment.

- As clarkb has mentioned, probably the most promising alternative is

to use the upstream container images as the basis for the initial chroot environments. jeblair has done most of this work with [3]. I'm fiddling with it to merge to master and see what's up ... I feel like maybe there were bootloader issues, although the basic extraction was working. This will allow the effort put into existing elements to not be lost.

If I had to pick; I'd probably say that using the nodepool-builder container is the best path. That has the most momentum behind it because it's used for the OpenDev image builds. As we work on the container-image base elements, this work will be deployed into the container (meaning the container is less reliant on the underlying version of Debian) and you can switch to them as appropriate.

I have to mention at this point, at risk of reharshing old debates, that an alternative in various scenarios (maybe not all) is the usage of libguestfs and its tools which modifies an existing base image.

https://libguestfs.org/

We switched to it in Sahara for most of the guest images and that saved some headaches when building from a different host. https://docs.openstack.org/sahara/latest/user/building-guest-images.html

I'd like to mention that libguestfs has been carrying a virt-dib tool for a while, but it has been tested only back to a certain version of dib: https://libguestfs.org/virt-dib.1.html

The major issues with libguestfs is that it seem primarily used to modify existing images. This has all of the problems that ianw points out above with size and stability. I'm sure you can bootstrap a base image to use with it too, but would you end up in the same toolchain problem as with dib if you did that?

Also, to be clear DIB supports the same use case of starting from an existing image (but using different tools) if you want to avoid the headaches with bootstrapping images yourself.

I think the more specific issue we're trying to figure out is "how do you bootstrap images from scratch for one distro on top of another if they use different bootstrapping tools". I don't think libguestfs helps with that much. And if you are starting from an existing image then DIB or libguestfs should work fine.

Uhm, maybe unattended virt-install could help there? -- Luigi

Hello again! Thanks everyone for joining the discussion and for the advice, much appreciated. To summarize the current situation: - at the moment it's not possible to build any rpm-based image on ubuntu focal using diskimage-builder with the centos-minimal element; - so far, we've successfully built the centos8 based ipa-ramdisk using diskimage-builder in ironic-python-agent-builder on centos-8 nodeset, but using the same nodeset also for testing the image (so running devstack) would be very time expensive because of the changes involved; - during the last ironic meeting, it has been decided to switch to the centos element and give that a try using the ubuntu focal nodeset; main concern about this approach is the final size of the ipa ramdisk, tests are ongoing and it looks promising: https://review.opendev.org/757808 https://review.opendev.org/757811 https://review.opendev.org/757812 Thanks again! Riccardo On Thu, Oct 8, 2020 at 6:11 PM Jeremy Stanley <fungi@yuggoth.org> wrote:

On 2020-10-08 17:57:55 +0200 (+0200), Luigi Toscano wrote: [...]

...

Uhm, maybe unattended virt-install could help there?

I expect that would be almost unworkable from within a virtual machine instance without nested virt acceleration. -- Jeremy Stanley