[ironic] IPA images: CentOS 8, Stream and Debian
Hi wonderful ironicers! Unless you spend the whole last week under a rock (good for you - I mean it!), you must be already aware that the classical CentOS as we know it is going away in favour of CentOS Stream, and maybe as soon as in the end of 2021! For us it means two things: 1) Bifrost will test with CentOS Stream, which may not 100% match the current stable RHEL. 2) We need to decide what to do with our published DIB images [1], currently based on CentOS 8. I'd like to concentrate on problem #2. We have conducted an experiment of switching to CentOS Stream, and it did not go so well. The image size has increased from ~ 340 MiB to nearly 450 MiB, causing serious issues in CI jobs using it. Because of this we're reverting the switch and considering other options. One is to figure out what causes the size increase and remove packages or manually delete files. We're already doing it quite intensively, so I expect this option to be time-consuming. We'll probably have to repeat this exercise regularly to keep up with the distribution changes. The other option is to switch to another distro. Debian looks promising in this context: 3 years of support and the image size is just 273 MiB. I have a patch [2] up for people who want to try the resulting image on their bare metal machines. What do you think? Are there any concerns with either option? [1] https://tarballs.openstack.org/ironic-python-agent/dib/files/ [2] https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/767158 -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
On 2020-12-15 17:08:16 +0100 (+0100), Dmitry Tantsur wrote: [...]
The other option is to switch to another distro. Debian looks promising in this context: 3 years of support and the image size is just 273 MiB. [...]
Not to be a Debian apologist, but depending on what you mean by "support" it could be usable for Ironic's purposes even longer: https://www.debian.org/lts/ The Zuul project and the OpenDev Collaboratory both use Debian as the basis of their container images too (though executed from Ubuntu virtual servers for now, in the latter case). -- Jeremy Stanley
On 12/15/20 5:08 PM, Dmitry Tantsur wrote:
Hi wonderful ironicers!
Unless you spend the whole last week under a rock (good for you - I mean it!), you must be already aware that the classical CentOS as we know it is going away in favour of CentOS Stream, and maybe as soon as in the end of 2021!
For us it means two things: 1) Bifrost will test with CentOS Stream, which may not 100% match the current stable RHEL. 2) We need to decide what to do with our published DIB images [1], currently based on CentOS 8.
I'd like to concentrate on problem #2.
We have conducted an experiment of switching to CentOS Stream, and it did not go so well. The image size has increased from ~ 340 MiB to nearly 450 MiB, causing serious issues in CI jobs using it. Because of this we're reverting the switch and considering other options.
One is to figure out what causes the size increase and remove packages or manually delete files. We're already doing it quite intensively, so I expect this option to be time-consuming. We'll probably have to repeat this exercise regularly to keep up with the distribution changes.
The other option is to switch to another distro. Debian looks promising in this context: 3 years of support and the image size is just 273 MiB. I have a patch [2] up for people who want to try the resulting image on their bare metal machines.
What do you think? Are there any concerns with either option?
[1] https://tarballs.openstack.org/ironic-python-agent/dib/files/ <https://tarballs.openstack.org/ironic-python-agent/dib/files/> [2] https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/767158 <https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/767158>
Hi Dmitry! I wonder what Debian image you've been using. Have you tried one of the daily image that the Debian Cloud Image Team prepares? [1] If you choose the Debian path, you have all of my support, and I'll try to help as much as I can. Cheers, Thomas Goirand (zigo) [1] http://cdimage.debian.org/cdimage/cloud/ and look for the "generic" images, which should work for Ironic.
On 2020-12-15 20:53:30 +0100 (+0100), Thomas Goirand wrote: [...]
I wonder what Debian image you've been using. Have you tried one of the daily image that the Debian Cloud Image Team prepares? [...]
While admitting that I don't really know much about it, I suspect "cloud" images would be unsuitable since what Ironic needs is really a RAMdisk to bootstrap bare metal server inventorying and deployment. -- Jeremy Stanley
On 12/15/20 9:22 PM, Jeremy Stanley wrote:
On 2020-12-15 20:53:30 +0100 (+0100), Thomas Goirand wrote: [...]
I wonder what Debian image you've been using. Have you tried one of the daily image that the Debian Cloud Image Team prepares? [...]
While admitting that I don't really know much about it, I suspect "cloud" images would be unsuitable since what Ironic needs is really a RAMdisk to bootstrap bare metal server inventorying and deployment.
You're assuming wrongly here. There's 2 types of images, one which we call "generic" that contains all the drivers, and one which we call "genericcloud" which uses the cloud kernel (ie: stripped with most hardware support, better suited for OpenStack Qemu hypervisor). The generic image is 284 MB, the cloud one is 221 MB (so 63 MB difference). So the "generic" image should be ok as an image for Ironic. As I never deployed Ironic myself, I'd be curious to know if the image is working well under that environment. Please let me know! Cheers, Thomas Goirand (zigo)
On Tue, Dec 15, 2020 at 8:56 PM Thomas Goirand <zigo@debian.org> wrote:
On 12/15/20 5:08 PM, Dmitry Tantsur wrote:
Hi wonderful ironicers!
Unless you spend the whole last week under a rock (good for you - I mean it!), you must be already aware that the classical CentOS as we know it is going away in favour of CentOS Stream, and maybe as soon as in the end of 2021!
For us it means two things: 1) Bifrost will test with CentOS Stream, which may not 100% match the current stable RHEL. 2) We need to decide what to do with our published DIB images [1], currently based on CentOS 8.
I'd like to concentrate on problem #2.
We have conducted an experiment of switching to CentOS Stream, and it did not go so well. The image size has increased from ~ 340 MiB to nearly 450 MiB, causing serious issues in CI jobs using it. Because of this we're reverting the switch and considering other options.
One is to figure out what causes the size increase and remove packages or manually delete files. We're already doing it quite intensively, so I expect this option to be time-consuming. We'll probably have to repeat this exercise regularly to keep up with the distribution changes.
The other option is to switch to another distro. Debian looks promising in this context: 3 years of support and the image size is just 273 MiB. I have a patch [2] up for people who want to try the resulting image on their bare metal machines.
What do you think? Are there any concerns with either option?
[1] https://tarballs.openstack.org/ironic-python-agent/dib/files/ <https://tarballs.openstack.org/ironic-python-agent/dib/files/> [2]
https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/767158
< https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/767158
Hi Dmitry!
I wonder what Debian image you've been using. Have you tried one of the daily image that the Debian Cloud Image Team prepares? [1]
Hi Thomas, We're using diskimage-builder, which, I think, uses debootstrap. This matches our requirements pretty well since we need as small images as it is possible.
If you choose the Debian path, you have all of my support, and I'll try to help as much as I can.
Thank you! Dmitry
Cheers,
Thomas Goirand (zigo)
[1] http://cdimage.debian.org/cdimage/cloud/ and look for the "generic" images, which should work for Ironic.
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
On 12/16/20 11:35 AM, Dmitry Tantsur wrote:
On Tue, Dec 15, 2020 at 8:56 PM Thomas Goirand <zigo@debian.org <mailto:zigo@debian.org>> wrote:
On 12/15/20 5:08 PM, Dmitry Tantsur wrote: > Hi wonderful ironicers! > > Unless you spend the whole last week under a rock (good for you - I mean > it!), you must be already aware that the classical CentOS as we know it > is going away in favour of CentOS Stream, and maybe as soon as in the > end of 2021! > > For us it means two things: > 1) Bifrost will test with CentOS Stream, which may not 100% match the > current stable RHEL. > 2) We need to decide what to do with our published DIB images [1], > currently based on CentOS 8. > > I'd like to concentrate on problem #2. > > We have conducted an experiment of switching to CentOS Stream, and it > did not go so well. The image size has increased from ~ 340 MiB to > nearly 450 MiB, causing serious issues in CI jobs using it. Because of > this we're reverting the switch and considering other options. > > One is to figure out what causes the size increase and remove packages > or manually delete files. We're already doing it quite intensively, so I > expect this option to be time-consuming. We'll probably have to repeat > this exercise regularly to keep up with the distribution changes. > > The other option is to switch to another distro. Debian looks promising > in this context: 3 years of support and the image size is just 273 MiB. > I have a patch [2] up for people who want to try the resulting image on > their bare metal machines. > > What do you think? Are there any concerns with either option? > > [1] https://tarballs.openstack.org/ironic-python-agent/dib/files/ <https://tarballs.openstack.org/ironic-python-agent/dib/files/> > <https://tarballs.openstack.org/ironic-python-agent/dib/files/ <https://tarballs.openstack.org/ironic-python-agent/dib/files/>> > [2] > https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/767158 <https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/767158> > <https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/767158 <https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/767158>>
Hi Dmitry!
I wonder what Debian image you've been using. Have you tried one of the daily image that the Debian Cloud Image Team prepares? [1]
Hi Thomas,
We're using diskimage-builder, which, I think, uses debootstrap. This matches our requirements pretty well since we need as small images as it is possible.
If you choose the Debian path, you have all of my support, and I'll try to help as much as I can.
Thank you!
Dmitry
Hi Dmitry, Could you please have a quick try with the official "Generic" image from the Debian team? Best maybe would be to try this one: http://cdimage.debian.org/cdimage/cloud/bullseye/daily/20201216-486/debian-1... Indeed, I would very much like to be able to validate the image works for Ironic. Cheers, Thomas Goirand (zigo)
On Wed, Dec 16, 2020 at 11:47 AM Thomas Goirand <zigo@debian.org> wrote:
On 12/16/20 11:35 AM, Dmitry Tantsur wrote:
On Tue, Dec 15, 2020 at 8:56 PM Thomas Goirand <zigo@debian.org <mailto:zigo@debian.org>> wrote:
On 12/15/20 5:08 PM, Dmitry Tantsur wrote: > Hi wonderful ironicers! > > Unless you spend the whole last week under a rock (good for you - I mean > it!), you must be already aware that the classical CentOS as we know it > is going away in favour of CentOS Stream, and maybe as soon as in
the
> end of 2021! > > For us it means two things: > 1) Bifrost will test with CentOS Stream, which may not 100% match
the
> current stable RHEL. > 2) We need to decide what to do with our published DIB images [1], > currently based on CentOS 8. > > I'd like to concentrate on problem #2. > > We have conducted an experiment of switching to CentOS Stream, and
it
> did not go so well. The image size has increased from ~ 340 MiB to > nearly 450 MiB, causing serious issues in CI jobs using it.
Because of
> this we're reverting the switch and considering other options. > > One is to figure out what causes the size increase and remove
packages
> or manually delete files. We're already doing it quite intensively, so I > expect this option to be time-consuming. We'll probably have to
repeat
> this exercise regularly to keep up with the distribution changes. > > The other option is to switch to another distro. Debian looks promising > in this context: 3 years of support and the image size is just 273 MiB. > I have a patch [2] up for people who want to try the resulting image on > their bare metal machines. > > What do you think? Are there any concerns with either option? > > [1] https://tarballs.openstack.org/ironic-python-agent/dib/files/ <https://tarballs.openstack.org/ironic-python-agent/dib/files/> > <https://tarballs.openstack.org/ironic-python-agent/dib/files/ <https://tarballs.openstack.org/ironic-python-agent/dib/files/>> > [2] >
https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/767158
<
https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/767158
> <
https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/767158
<
Hi Dmitry!
I wonder what Debian image you've been using. Have you tried one of
https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/767158 the
daily image that the Debian Cloud Image Team prepares? [1]
Hi Thomas,
We're using diskimage-builder, which, I think, uses debootstrap. This matches our requirements pretty well since we need as small images as it is possible.
If you choose the Debian path, you have all of my support, and I'll
try
to help as much as I can.
Thank you!
Dmitry
Hi Dmitry,
Could you please have a quick try with the official "Generic" image from the Debian team? Best maybe would be to try this one:
http://cdimage.debian.org/cdimage/cloud/bullseye/daily/20201216-486/debian-1...
Only if DIB provides support for it. I'm not inclined to reproduce the whole generation process manually. Dmitry
Indeed, I would very much like to be able to validate the image works for Ironic.
Cheers,
Thomas Goirand (zigo)
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
On 12/16/20 11:59 AM, Dmitry Tantsur wrote:
On Wed, Dec 16, 2020 at 11:47 AM Thomas Goirand <zigo@debian.org Hi Dmitry,
Could you please have a quick try with the official "Generic" image from the Debian team? Best maybe would be to try this one:
http://cdimage.debian.org/cdimage/cloud/bullseye/daily/20201216-486/debian-1... <http://cdimage.debian.org/cdimage/cloud/bullseye/daily/20201216-486/debian-11-generic-amd64-daily-20201216-486.qcow2>
Only if DIB provides support for it. I'm not inclined to reproduce the whole generation process manually.
Dmitry
Hi Dmitry, I'm not asking that you switch within the CI, I'm just asking if you can just try the Debian image *once* and *anywhere* you like, just to confirm that the image works. Anyone else volunteering would to do... :) Cheers, Thomas Goirand (zigo)
I think there's still a basic disconnect. Ironic builds a ramdisk image from scratch, currently using DIB, to run IPA in an ephemeral environment for provisioning and cleaning. We can't run just a simple qcow image. I think the current methods using debootstrap are great because it makes it trivial to setup a new image using DIB. If we were to use published debian images, we'd have to edit them, embed the IPA ramdisk, and ensure they're in a separate kernel:ramdisk format to be used as a ramdisk. In my experience, that's pretty difficult to do with a standard cloud image. - Jay Faulkner On Wed, Dec 16, 2020 at 7:29 AM Thomas Goirand <zigo@debian.org> wrote:
On 12/16/20 11:59 AM, Dmitry Tantsur wrote:
On Wed, Dec 16, 2020 at 11:47 AM Thomas Goirand <zigo@debian.org Hi Dmitry,
Could you please have a quick try with the official "Generic" image from the Debian team? Best maybe would be to try this one:
<
Only if DIB provides support for it. I'm not inclined to reproduce the whole generation process manually.
Dmitry
Hi Dmitry,
I'm not asking that you switch within the CI, I'm just asking if you can just try the Debian image *once* and *anywhere* you like, just to confirm that the image works. Anyone else volunteering would to do... :)
Cheers,
Thomas Goirand (zigo)
participants (4)
-
Dmitry Tantsur
-
Jay Faulkner
-
Jeremy Stanley
-
Thomas Goirand