[qa][devstack] Moving to release specific nodesets and jobs for Fedora
Hello all, $subject has come up with the last couple rounds of changes introducing support for the latest version of Fedora within devstack. I've pushed a change introducing version specific nodesets and jobs for Fedora 31 as a way of starting a discussion around this ahead of the PTG: WIP Fedora: Start using version specific nodesets and jobs https://review.opendev.org/#/c/719775/ I'm personally on the fence about this as while it would be nice to have version specific nodesets and jobs it does introduce a lot of churn with each new release of Fedora across many projects: http://codesearch.openstack.org/?q=fedora%5C-latest Cheers, -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
On Tue, Apr 14, 2020 at 10:38:26AM +0100, Lee Yarwood wrote:
I'm personally on the fence about this as while it would be nice to have version specific nodesets and jobs it does introduce a lot of churn with each new release of Fedora across many projects:
This was exactly why fedora-latest was introduced; so that on new distro releases all those got bumped automatically. The idea was that if you're using Fedora you basically expect a bump to happen for you. Sometimes we get Fedora updates in within days of release, and other times it takes *a lot* longer. F31 has taken *a lot* longer; because it stopped being able to be built on our extant Xenial builder hosts because it moved packages to a new form of compression not supported by the RPM available there. Ergo we couldn't extract the rpms to the base chroot that everything else builds ontop of. So basically it got put behind creating/debugging nodepool-builder container images, rewriting testing for these container images and figuring out how to deploy them to production. That is done now with our nb04 host where Fedora is building now. We know that the "run other distro tools on the host" approach is fragile; using containers isolates us from host changes pretty well now, but longer term there's ideas in progress around building up our CI images from containers too. So I don't feel like we want to go back to having a big pain bumping Fedora releases each time; we should hopefully be in a position to push out future releases in a much more timely manner. That said ... Fedora never ceases to surprise! -i
On 15-04-20 06:10:59, Ian Wienand wrote:
On Tue, Apr 14, 2020 at 10:38:26AM +0100, Lee Yarwood wrote:
I'm personally on the fence about this as while it would be nice to have version specific nodesets and jobs it does introduce a lot of churn with each new release of Fedora across many projects:
This was exactly why fedora-latest was introduced; so that on new distro releases all those got bumped automatically. The idea was that if you're using Fedora you basically expect a bump to happen for you.
Sometimes we get Fedora updates in within days of release, and other times it takes *a lot* longer. F31 has taken *a lot* longer; because it stopped being able to be built on our extant Xenial builder hosts because it moved packages to a new form of compression not supported by the RPM available there. Ergo we couldn't extract the rpms to the base chroot that everything else builds ontop of.
So basically it got put behind creating/debugging nodepool-builder container images, rewriting testing for these container images and figuring out how to deploy them to production. That is done now with our nb04 host where Fedora is building now.
We know that the "run other distro tools on the host" approach is fragile; using containers isolates us from host changes pretty well now, but longer term there's ideas in progress around building up our CI images from containers too.
Understood, going off on a slight tangent here but has virt-builder ever been considered for building these CI images? I appreciate there's a huge amount of logic built up in the dib elements at this point but I've always assumed you could inject that into a virt-builder run somehow.
So I don't feel like we want to go back to having a big pain bumping Fedora releases each time; we should hopefully be in a position to push out future releases in a much more timely manner.
That said ... Fedora never ceases to surprise!
ACK thanks Ian, I'll close my version specific change out. Cheers, -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
Hey, I agree in that under current circumstances Fedora does not fit well the model of other stabler distros which have their node labels named after release. So let's not do this for now. -yoctozepto czw., 16 kwi 2020 o 11:20 Lee Yarwood <lyarwood@redhat.com> napisał(a):
On 15-04-20 06:10:59, Ian Wienand wrote:
On Tue, Apr 14, 2020 at 10:38:26AM +0100, Lee Yarwood wrote:
I'm personally on the fence about this as while it would be nice to have version specific nodesets and jobs it does introduce a lot of churn with each new release of Fedora across many projects:
This was exactly why fedora-latest was introduced; so that on new distro releases all those got bumped automatically. The idea was that if you're using Fedora you basically expect a bump to happen for you.
Sometimes we get Fedora updates in within days of release, and other times it takes *a lot* longer. F31 has taken *a lot* longer; because it stopped being able to be built on our extant Xenial builder hosts because it moved packages to a new form of compression not supported by the RPM available there. Ergo we couldn't extract the rpms to the base chroot that everything else builds ontop of.
So basically it got put behind creating/debugging nodepool-builder container images, rewriting testing for these container images and figuring out how to deploy them to production. That is done now with our nb04 host where Fedora is building now.
We know that the "run other distro tools on the host" approach is fragile; using containers isolates us from host changes pretty well now, but longer term there's ideas in progress around building up our CI images from containers too.
Understood, going off on a slight tangent here but has virt-builder ever been considered for building these CI images? I appreciate there's a huge amount of logic built up in the dib elements at this point but I've always assumed you could inject that into a virt-builder run somehow.
So I don't feel like we want to go back to having a big pain bumping Fedora releases each time; we should hopefully be in a position to push out future releases in a much more timely manner.
That said ... Fedora never ceases to surprise!
ACK thanks Ian, I'll close my version specific change out.
Cheers,
-- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
On 15-04-20 06:10:59, Ian Wienand wrote:
On Tue, Apr 14, 2020 at 10:38:26AM +0100, Lee Yarwood wrote:
I'm personally on the fence about this as while it would be nice to have version specific nodesets and jobs it does introduce a lot of churn with each new release of Fedora across many projects:
This was exactly why fedora-latest was introduced; so that on new distro releases all those got bumped automatically. The idea was that if you're using Fedora you basically expect a bump to happen for you.
Sometimes we get Fedora updates in within days of release, and other times it takes *a lot* longer. F31 has taken *a lot* longer; because it stopped being able to be built on our extant Xenial builder hosts because it moved packages to a new form of compression not supported by the RPM available there. Ergo we couldn't extract the rpms to the base chroot that everything else builds ontop of.
So basically it got put behind creating/debugging nodepool-builder container images, rewriting testing for these container images and figuring out how to deploy them to production. That is done now with our nb04 host where Fedora is building now.
We know that the "run other distro tools on the host" approach is fragile; using containers isolates us from host changes pretty well now, but longer term there's ideas in progress around building up our CI images from containers too.
Understood, going off on a slight tangent here but has virt-builder ever been considered for building these CI images? I appreciate there's a huge amount of logic built up in the dib elements at this point but I've always assumed you could inject that into a virt-builder run somehow. virtbuild does require you to have qemu installed among other deps which are not currently required by dib and honestly i have always had much better experince using dib the virtbuilder. the creattion an dreuse of dib element is relitivly simple and virt builder itself is not really set up to allow you od do that your self. sure there are templates for different distro and you could proably make your own but virt-build cant make image for diferent targets such as docker images as far as i am aware
On Thu, 2020-04-16 at 10:11 +0100, Lee Yarwood wrote: like dib can so i dont think its really a viable alternitive. i mean it could be with work to port everything but im not sure its worth it.
So I don't feel like we want to go back to having a big pain bumping Fedora releases each time; we should hopefully be in a position to push out future releases in a much more timely manner.
That said ... Fedora never ceases to surprise!
ACK thanks Ian, I'll close my version specific change out.
Cheers,
On Thu, Apr 16, 2020 at 10:11:47AM +0100, Lee Yarwood wrote:
Understood, going off on a slight tangent here but has virt-builder ever been considered for building these CI images? I appreciate there's a huge amount of logic built up in the dib elements at this point but I've always assumed you could inject that into a virt-builder run somehow.
Not seriously -- for a start I'm yet to find a tool that can build *everything* we're interested in; including centos, fedora, debian, suse and gentoo; despite [1] being impressive it still lacks coverage of some of them. As you say, most of the logic is in the various elements doing customisation. If anything the trend is to *decrease* the logic there and move it into Zuul base jobs -- there is a lot of history, and the amazing templating and inheritance that we have in our jobs now was not always available, leading to some of the choices we made at the time to do various things to the images we make available which we would not do now. There is some interest in making dib builds bootstrap themselves from containers; see [2] for a WIP change. This avoids some of the issues with requiring the host to have tools to build guest chroots. -i [1] http://builder.libguestfs.org/index.asc [2] https://review.opendev.org/#/c/700083/
participants (4)
-
Ian Wienand
-
Lee Yarwood
-
Radosław Piliszek
-
Sean Mooney