[all] [kolla] Kolla Builder - next generation
Hello fellow OpenStackers and Kollars (or Koalas) in particular, today is the day when I finally sit down to write up my thoughts on Kolla, and I mean *Kolla*, the container image building project for OpenStack and friends, nothing more (nor less). *** Some background on Kolla to get everyone on the same page (or close enough): Kolla builds images for production use. Kolla is upstream for TripleO containers and Kolla-Ansible deployment. These are container images, think OCI and Docker in particular since Kolla actually relies on Dockerfile format to specify build recipes and Docker runs building (TripleO runs buildah on the same recipes). Kolla supports three distributions as bases: CentOS (TripleO does override this for RHEL as well), Debian and Ubuntu. In the interim periods Kolla supports two distribution releases to ease/smooth the transition process for operators (like currently CentOS 7 and 8 in Train, while Ussuri is 8 only). What is more, each distro has two flavours (or 'types' as they are called for now): binary and source. This 'type' applies only to OpenStack software. Binary means that Kolla uses downstream packages (rpm/deb) while source means to use pip and install from official tarballs (or repos if it is master) and PyPI for deps, utilising venv to keep it separate from distro stuff. Finally, there is particular target architecture: x86_64, aarch64 and ppc64le. All of above affect the support matrix which is based on 3 or 4 dimensions (distro, flavour, arch and sometimes distro version). [1] Kolla offers a high level of customisability via various overrides levels. See docs for details [2] Kolla engine offers hierarchical approach to image building, under the assumption that more than one image is often deployed on the same machine so layer sharing is beneficial. Kolla helps collect sources for OpenStack projects to build desired versions of them. All recipes are templated using Jinja2 syntax. Images contain both run-time and, mostly in the case of 'source' flavour, build-time tools and libraries. *** That's it for the background. :-) So where is the problem you might ask? Oh, there are plenty. The general is that Kolla has lots of logic in templates. Thankfully, Kolla has macros for most stuff but still Jinja2 limitations make it hard to document exceptions (no inline comments in arrays anyone?). We lose visibility into real dependency graphs and may be easily reinstalling same stuff, any optimisations are limited and would require parsing of both Jinja2 and Dockerfile syntax. In general the current approach is ugly and becomes unwieldy (e.g. getting warnings about empty continuation lines). There is layering hell: dependency on long &&ed commands which are templated out to avoid useless layers. Support matrix is great for an overview. There is a hidden layer to it though, different combinations may support different features. This is not documented so far and not so easy to follow from sources. This stems from the fact that different extra components have different availability in distros, but this is not easily apparent from sources. Also, there is no way to turn feature on/off. You might do an override, but then again, which to keep, which not to? Which is which? Oh! Finally, the images tend to be heavy due to inclusion of build-time deps. *** That's it for issues (hopefully!) My idea is to reuse Kolla engine where it shines: sources collection, system of plugins, hierarchical building; but replace the part that smells - Jinja2 templating. Giving up on Jinja2 might encompass giving up on Dockerfile syntax, but that is optional and depends on what makes it more amenable to avoid further pitfalls. With giving up on Jinja2, the idea is to generate building recipes fully programmatically from Python. It would be possible to introduce "Features" - sets of packages to install based on the (distro, arch) tuple. This would result in more flexibility - turning them on/off (some could be optional, some not). There could be more than one optimization strategy regarding when packages get installed: you want only standalone blah-blah? Then Kolla won't be installing XYZ and ABC just because ugma-ugma and tickle-tickle require them and you "could save some space" (TM). In the same vein, features could declare which components are build-time and which are run-time and this would make it straightforward to separate the sides. The above effort could well be coordinated with different projects to reuse bindep contents. So far Kolla does not use bindep because it often installs too much and not enough at the same time. Do note it would still be bindep-less for external services. There would still need to exist a general mechanism for providing custom command executions required by some images. For contributors and cores this new approach would bring more sanity as to the scope of proposed changes. Also, it would be possible to get quick insight into feature support and autogenerate docs for that as well. Similarly, current concept of unbuildable images would no longer be required because unbuildability would be dictated by lack of support for a required feature. *** Looking forward to your opinions/thoughts. *** [1] https://docs.openstack.org/kolla/train/support_matrix.html [2] https://docs.openstack.org/kolla/train/admin/image-building.html -yoctozepto
Hello fellow OpenStackers and Kollars (or Koalas) in particular,
today is the day when I finally sit down to write up my thoughts on Kolla, and I mean *Kolla*, the container image building project for OpenStack and friends, nothing more (nor less).
***
Some background on Kolla to get everyone on the same page (or close enough):
Kolla builds images for production use. Kolla is upstream for TripleO containers and Kolla-Ansible deployment. These are container images, think OCI and Docker in particular since Kolla actually relies on Dockerfile format to specify build recipes and Docker runs building (TripleO runs buildah on the same recipes).
Kolla supports three distributions as bases: CentOS (TripleO does override this for RHEL as well), Debian and Ubuntu. In the interim periods Kolla supports two distribution releases to ease/smooth the transition process for operators (like currently CentOS 7 and 8 in Train, while Ussuri is 8 only).
What is more, each distro has two flavours (or 'types' as they are called for now): binary and source. This 'type' applies only to OpenStack software. Binary means that Kolla uses downstream packages (rpm/deb) while source means to use pip and install from official tarballs (or repos if it is master) and PyPI for deps, utilising venv to keep it separate from distro stuff.
Finally, there is particular target architecture: x86_64, aarch64 and ppc64le.
All of above affect the support matrix which is based on 3 or 4 dimensions (distro, flavour, arch and sometimes distro version). [1]
Kolla offers a high level of customisability via various overrides levels. See docs for details [2]
Kolla engine offers hierarchical approach to image building, under the assumption that more than one image is often deployed on the same machine so layer sharing is beneficial. Kolla helps collect sources for OpenStack projects to build desired versions of them.
All recipes are templated using Jinja2 syntax.
Images contain both run-time and, mostly in the case of 'source' flavour, build-time tools and libraries.
*** That's it for the background. :-)
So where is the problem you might ask? Oh, there are plenty.
The general is that Kolla has lots of logic in templates. Thankfully, Kolla has macros for most stuff but still Jinja2 limitations make it hard to document exceptions (no inline comments in arrays anyone?). We lose visibility into real dependency graphs and may be easily reinstalling same stuff, any optimisations are limited and would require parsing of both Jinja2 and Dockerfile syntax. In general the current approach is ugly and becomes unwieldy (e.g. getting warnings about empty continuation lines). There is layering hell: dependency on long &&ed commands which are templated out to avoid useless layers.
Support matrix is great for an overview. There is a hidden layer to it though, different combinations may support different features. This is not documented so far and not so easy to follow from sources. This stems from the fact that different extra components have different availability in distros, but this is not easily apparent from sources. Also, there is no way to turn feature on/off. You might do an override, but then again, which to keep, which not to? Which is which? Oh!
Finally, the images tend to be heavy due to inclusion of build-time deps.
*** That's it for issues (hopefully!)
My idea is to reuse Kolla engine where it shines: sources collection, system of plugins, hierarchical building; but replace the part that smells - Jinja2 templating. i acutlly tought that the use of jinja2 was one of the nicer parts of how we build images. its been a long time since i was activly looking at kolla but the fact that jinja2 was used both in the image building and in the config generation in kolla-ansible was nice as i did not have to learn a new syntax or mechanisium for each and could transfer knoladge between them. Giving up on Jinja2 might encompass giving up on Dockerfile syntax, but that is optional and depends on what makes it more amenable to avoid further pitfalls.
With giving up on Jinja2, the idea is to generate building recipes fully programmatically from Python.
It would be possible to introduce "Features" - sets of packages to install based on the (distro, arch) tuple. This would result in more flexibility - turning them on/off (some could be optional, some not). There could be more than one optimization strategy regarding when packages get installed: you want only standalone blah-blah? Then Kolla won't be installing XYZ and ABC just because ugma-ugma and tickle-tickle require them and you "could save some space" (TM). In the same vein, features could declare which components are build-time and which are run-time and this would make it straightforward to separate the sides. The above effort could well be coordinated with different projects to reuse bindep contents. So far Kolla does not use bindep because it often installs too much and not enough at the same time. Do note it would still be bindep-less for external services.
On Sat, 2020-04-18 at 16:09 +0200, Radosław Piliszek wrote: this woudl break our templeate override feature which is how ooo customises the images for rhel. there was a effort in the past to drop using docker for the building of the images to achive a similar goal. specifically the creation of a DSL or a alternitve builder that does nto use layers https://review.opendev.org/#/c/503882/ using either approch woudl allow us to keep jinja/tempelate overrides while perhspas still reducing the size of iamges and adressin the build time vs runtime deps issue.
There would still need to exist a general mechanism for providing custom command executions required by some images.
For contributors and cores this new approach would bring more sanity as to the scope of proposed changes. Also, it would be possible to get quick insight into feature support and autogenerate docs for that as well. Similarly, current concept of unbuildable images would no longer be required because unbuildability would be dictated by lack of support for a required feature.
***
Looking forward to your opinions/thoughts.
***
[1] https://docs.openstack.org/kolla/train/support_matrix.html [2] https://docs.openstack.org/kolla/train/admin/image-building.html
-yoctozepto
On Mon, Apr 20, 2020 at 2:29 PM Sean Mooney <smooney@redhat.com> wrote:
On Sat, 2020-04-18 at 16:09 +0200, Radosław Piliszek wrote:
My idea is to reuse Kolla engine where it shines: sources collection, system of plugins, hierarchical building; but replace the part that smells - Jinja2 templating. i acutlly tought that the use of jinja2 was one of the nicer parts of how we build images. its been a long time since i was activly looking at kolla but the fact that jinja2 was used both in the image building and in the config generation in kolla-ansible was nice as i did not have to learn a new syntax or mechanisium for each and could transfer knoladge between them.
It's nice as long as you don't do anything fancy and don't want to run analysis on them. The more we rely on jinja2, the more we rely on humans to verify everything. Unit testing is non-existent. Under the new model, whatever was easy with jinja2 wouldn't be harder, and what was convoluted, would no longer be (that much). (Self-)documenting reliatonships between packages is a nice way to ensure we bring similar experience regardless of final platform. With current conditionals hell it's unimaginable to handle other than by further functional and integration testing (kolla-ansible).
Giving up on Jinja2 might encompass giving up on Dockerfile syntax, but that is optional and depends on what makes it more amenable to avoid further pitfalls.
With giving up on Jinja2, the idea is to generate building recipes fully programmatically from Python. this woudl break our templeate override feature which is how ooo customises the images for rhel.
Compatibility layer would be in place. Kolla already coordinates changes with TripleO and we would keep that in check.
It would be possible to introduce "Features" - sets of packages to install based on the (distro, arch) tuple. This would result in more flexibility - turning them on/off (some could be optional, some not). There could be more than one optimization strategy regarding when packages get installed: you want only standalone blah-blah? Then Kolla won't be installing XYZ and ABC just because ugma-ugma and tickle-tickle require them and you "could save some space" (TM). In the same vein, features could declare which components are build-time and which are run-time and this would make it straightforward to separate the sides. The above effort could well be coordinated with different projects to reuse bindep contents. So far Kolla does not use bindep because it often installs too much and not enough at the same time. Do note it would still be bindep-less for external services. there was a effort in the past to drop using docker for the building of the images to achive a similar goal. specifically the creation of a DSL or a alternitve builder that does nto use layers https://review.opendev.org/#/c/503882/ using either approch woudl allow us to keep jinja/tempelate overrides while perhspas still reducing the size of iamges and adressin the build time vs runtime deps issue.
Thanks for the link, might come in handy, starred and added to my queue. -yoctozepto
On Sat, 18 Apr 2020 at 15:10, Radosław Piliszek <radoslaw.piliszek@gmail.com> wrote:
Hello fellow OpenStackers and Kollars (or Koalas) in particular,
today is the day when I finally sit down to write up my thoughts on Kolla, and I mean *Kolla*, the container image building project for OpenStack and friends, nothing more (nor less).
Thanks for writing this up Radosław. It's good to engage in some open ended proposals and discussion from time to time.
***
Some background on Kolla to get everyone on the same page (or close enough):
Kolla builds images for production use. Kolla is upstream for TripleO containers and Kolla-Ansible deployment. These are container images, think OCI and Docker in particular since Kolla actually relies on Dockerfile format to specify build recipes and Docker runs building (TripleO runs buildah on the same recipes).
Kolla supports three distributions as bases: CentOS (TripleO does override this for RHEL as well), Debian and Ubuntu. In the interim periods Kolla supports two distribution releases to ease/smooth the transition process for operators (like currently CentOS 7 and 8 in Train, while Ussuri is 8 only).
What is more, each distro has two flavours (or 'types' as they are called for now): binary and source. This 'type' applies only to OpenStack software. Binary means that Kolla uses downstream packages (rpm/deb) while source means to use pip and install from official tarballs (or repos if it is master) and PyPI for deps, utilising venv to keep it separate from distro stuff.
Finally, there is particular target architecture: x86_64, aarch64 and ppc64le.
All of above affect the support matrix which is based on 3 or 4 dimensions (distro, flavour, arch and sometimes distro version). [1]
Kolla offers a high level of customisability via various overrides levels. See docs for details [2]
Kolla engine offers hierarchical approach to image building, under the assumption that more than one image is often deployed on the same machine so layer sharing is beneficial.
It also makes the build more efficient if you build multiple images at once.
Kolla helps collect sources for OpenStack projects to build desired versions of them.
All recipes are templated using Jinja2 syntax.
Images contain both run-time and, mostly in the case of 'source' flavour, build-time tools and libraries.
*** That's it for the background. :-)
So where is the problem you might ask? Oh, there are plenty.
The general is that Kolla has lots of logic in templates. Thankfully, Kolla has macros for most stuff but still Jinja2 limitations make it hard to document exceptions (no inline comments in arrays anyone?). We lose visibility into real dependency graphs and may be easily reinstalling same stuff, any optimisations are limited and would require parsing of both Jinja2 and Dockerfile syntax. In general the current approach is ugly and becomes unwieldy (e.g. getting warnings about empty continuation lines). There is layering hell: dependency on long &&ed commands which are templated out to avoid useless layers.
Support matrix is great for an overview. There is a hidden layer to it though, different combinations may support different features. This is not documented so far and not so easy to follow from sources. This stems from the fact that different extra components have different availability in distros, but this is not easily apparent from sources. Also, there is no way to turn feature on/off. You might do an override, but then again, which to keep, which not to? Which is which? Oh!
Finally, the images tend to be heavy due to inclusion of build-time deps.
My multi-stage build PoC [1] suggests about 150MB could be saved for source images. With shared image layers we only get that saving once, in openstack-base, so this would have most effect in an environment where images are built individually at different times, and may not share layers.
*** That's it for issues (hopefully!)
My idea is to reuse Kolla engine where it shines: sources collection, system of plugins, hierarchical building; but replace the part that smells - Jinja2 templating. Giving up on Jinja2 might encompass giving up on Dockerfile syntax, but that is optional and depends on what makes it more amenable to avoid further pitfalls.
With giving up on Jinja2, the idea is to generate building recipes fully programmatically from Python.
There would be some nice advantages to this, not least it would be Python rather than Jinja. Also it would be nicely unit testable. However, Jinja does give us some nice properties. Dockerfiles are relatively WYSIWIG, and this keeps the barrier to contribution low. Often the most that needs to be done to help a contributor get started with kolla is to say 'here is the Dockerfile.j2 for the image you want to change'. As Sean rightly pointed out, we have wedded ourselves to Jinja by exposing it as a customisation point, through blocks and overrides. We'd need to replicate that. A potential substitute for unit testing could be to generate every supported combination of Dockerfiles, and keep them in the repository. These would need to be updated with each change, but would allow us to see the final content of the images for each distro. Even better than this would be to use TestInfra [2] to validate various expectations about image content (e.g. virtualenv uses py3.6, package foo is installed).
It would be possible to introduce "Features" - sets of packages to install based on the (distro, arch) tuple. This would result in more flexibility - turning them on/off (some could be optional, some not). There could be more than one optimization strategy regarding when packages get installed: you want only standalone blah-blah? Then Kolla won't be installing XYZ and ABC just because ugma-ugma and tickle-tickle require them and you "could save some space" (TM). In the same vein, features could declare which components are build-time and which are run-time and this would make it straightforward to separate the sides.
This could be neat, although it would be difficult to test and keep working, and would only increase our support matrix. You could do it with Jinja, although it might get a bit unwieldy.
The above effort could well be coordinated with different projects to reuse bindep contents. So far Kolla does not use bindep because it often installs too much and not enough at the same time. Do note it would still be bindep-less for external services.
I like the idea of using bindep for source images, in combination with multi-stage builds. If we could reduce the length of our package dependency lists by relying on bindep, that would be a nice win. Ideally, between distro package dependencies and bindep, we would need to specify few additional packages.
There would still need to exist a general mechanism for providing custom command executions required by some images.
For contributors and cores this new approach would bring more sanity as to the scope of proposed changes. Also, it would be possible to get quick insight into feature support and autogenerate docs for that as well. Similarly, current concept of unbuildable images would no longer be required because unbuildability would be dictated by lack of support for a required feature.
I'm not sure what you mean by 'feature' here, but I'd say the most common reason an image is unbuildable currently is that the main package required by the image is unavailable, rather than some ancillary package required for a particular feature. Modifying unbuildable images to include unavailable features could be interesting, but would be an extension of current behaviour.
***
Looking forward to your opinions/thoughts.
I think there are some good ideas in here, but I feel that the cost/benefit of the core proposal to replace Jinja with Python doesn't work for me. Specifically, the cost would be high in terms of being a significant rewrite of every image and building additional tooling, plus the operator headaches of switching to the new customisation model. In terms of benefits, it seems they are mostly for contributors rather than users. I'd like us to explore some of the pain points raised here, and also see if we can determine any others through the kolla klub. * Can we reduce the image size? Are we installing unnecessary packages? (spoiler, yes) Can we use multi stage builds? * Can we improve testing, to make it clearer what the effects of changing a particular image would be? * What would a 'feature' look like with our current tooling? I'd like to see a concrete example. * What can we learn from the proposed goal [3] to add container images for each project? Could kolla be used in a more distributed manner more amenable to a CI/CD pattern where each project publishes its own images? Could we add a 'python' base distro that is based on the python:3-slim image? I realise the authors of the goal won't like this, but it does add some missing flexibility to their proposal. [1] https://review.opendev.org/#/c/631647/ [2] https://testinfra.readthedocs.io/en/latest/ [3] https://review.opendev.org/#/c/720107/
***
[1] https://docs.openstack.org/kolla/train/support_matrix.html [2] https://docs.openstack.org/kolla/train/admin/image-building.html
-yoctozepto
On Tue, Apr 21, 2020 at 5:15 AM Mark Goddard <mark@stackhpc.com> wrote:
On Sat, 18 Apr 2020 at 15:10, Radosław Piliszek <radoslaw.piliszek@gmail.com> wrote:
Hello fellow OpenStackers and Kollars (or Koalas) in particular,
today is the day when I finally sit down to write up my thoughts on Kolla, and I mean *Kolla*, the container image building project for OpenStack and friends, nothing more (nor less).
Thanks for writing this up Radosław. It's good to engage in some open ended proposals and discussion from time to time.
***
Some background on Kolla to get everyone on the same page (or close enough):
Kolla builds images for production use. Kolla is upstream for TripleO containers and Kolla-Ansible deployment. These are container images, think OCI and Docker in particular since Kolla actually relies on Dockerfile format to specify build recipes and Docker runs building (TripleO runs buildah on the same recipes).
Kolla supports three distributions as bases: CentOS (TripleO does override this for RHEL as well), Debian and Ubuntu. In the interim periods Kolla supports two distribution releases to ease/smooth the transition process for operators (like currently CentOS 7 and 8 in Train, while Ussuri is 8 only).
What is more, each distro has two flavours (or 'types' as they are called for now): binary and source. This 'type' applies only to OpenStack software. Binary means that Kolla uses downstream packages (rpm/deb) while source means to use pip and install from official tarballs (or repos if it is master) and PyPI for deps, utilising venv to keep it separate from distro stuff.
Finally, there is particular target architecture: x86_64, aarch64 and ppc64le.
All of above affect the support matrix which is based on 3 or 4 dimensions (distro, flavour, arch and sometimes distro version). [1]
Kolla offers a high level of customisability via various overrides levels. See docs for details [2]
Kolla engine offers hierarchical approach to image building, under the assumption that more than one image is often deployed on the same machine so layer sharing is beneficial.
It also makes the build more efficient if you build multiple images at once.
Kolla helps collect sources for OpenStack projects to build desired versions of them.
All recipes are templated using Jinja2 syntax.
Images contain both run-time and, mostly in the case of 'source' flavour, build-time tools and libraries.
*** That's it for the background. :-)
So where is the problem you might ask? Oh, there are plenty.
The general is that Kolla has lots of logic in templates. Thankfully, Kolla has macros for most stuff but still Jinja2 limitations make it hard to document exceptions (no inline comments in arrays anyone?). We lose visibility into real dependency graphs and may be easily reinstalling same stuff, any optimisations are limited and would require parsing of both Jinja2 and Dockerfile syntax. In general the current approach is ugly and becomes unwieldy (e.g. getting warnings about empty continuation lines). There is layering hell: dependency on long &&ed commands which are templated out to avoid useless layers.
Support matrix is great for an overview. There is a hidden layer to it though, different combinations may support different features. This is not documented so far and not so easy to follow from sources. This stems from the fact that different extra components have different availability in distros, but this is not easily apparent from sources. Also, there is no way to turn feature on/off. You might do an override, but then again, which to keep, which not to? Which is which? Oh!
Finally, the images tend to be heavy due to inclusion of build-time deps.
My multi-stage build PoC [1] suggests about 150MB could be saved for source images. With shared image layers we only get that saving once, in openstack-base, so this would have most effect in an environment where images are built individually at different times, and may not share layers.
*** That's it for issues (hopefully!)
My idea is to reuse Kolla engine where it shines: sources collection, system of plugins, hierarchical building; but replace the part that smells - Jinja2 templating. Giving up on Jinja2 might encompass giving up on Dockerfile syntax, but that is optional and depends on what makes it more amenable to avoid further pitfalls.
With giving up on Jinja2, the idea is to generate building recipes fully programmatically from Python.
There would be some nice advantages to this, not least it would be Python rather than Jinja. Also it would be nicely unit testable.
However, Jinja does give us some nice properties. Dockerfiles are relatively WYSIWIG, and this keeps the barrier to contribution low. Often the most that needs to be done to help a contributor get started with kolla is to say 'here is the Dockerfile.j2 for the image you want to change'.
As Sean rightly pointed out, we have wedded ourselves to Jinja by exposing it as a customisation point, through blocks and overrides. We'd need to replicate that.
A potential substitute for unit testing could be to generate every supported combination of Dockerfiles, and keep them in the repository. These would need to be updated with each change, but would allow us to see the final content of the images for each distro. Even better than this would be to use TestInfra [2] to validate various expectations about image content (e.g. virtualenv uses py3.6, package foo is installed).
It would be possible to introduce "Features" - sets of packages to install based on the (distro, arch) tuple. This would result in more flexibility - turning them on/off (some could be optional, some not). There could be more than one optimization strategy regarding when packages get installed: you want only standalone blah-blah? Then Kolla won't be installing XYZ and ABC just because ugma-ugma and tickle-tickle require them and you "could save some space" (TM). In the same vein, features could declare which components are build-time and which are run-time and this would make it straightforward to separate the sides.
This could be neat, although it would be difficult to test and keep working, and would only increase our support matrix. You could do it with Jinja, although it might get a bit unwieldy.
The above effort could well be coordinated with different projects to reuse bindep contents. So far Kolla does not use bindep because it often installs too much and not enough at the same time. Do note it would still be bindep-less for external services.
I like the idea of using bindep for source images, in combination with multi-stage builds. If we could reduce the length of our package dependency lists by relying on bindep, that would be a nice win. Ideally, between distro package dependencies and bindep, we would need to specify few additional packages.
I think this is a golden ticket of what I'm trying to accomplish and this is what OpenDev is currently doing. Ideally, there _should_ be no distro package dependencies because they should all be modeled inside bindep (using different profiles if needed). I still think that if we do this, there is a lot of value in being able to do this in-repo because those Dockerfile's will be pretty static. While I appreciate that Kolla has all sorts of neat things to customize images, for a certain operator base, they'd much rather either have a Dockerfile to build off of or an existing base image somewhere. Kolla could be the thing that drives this, Kolla could become "OpenStack containers", or parts of it could be. However, I think the "customization bits" should ideally stay outside of the scope of this, because we just want to publish an artifact with dependencies installed.
There would still need to exist a general mechanism for providing custom command executions required by some images.
For contributors and cores this new approach would bring more sanity as to the scope of proposed changes. Also, it would be possible to get quick insight into feature support and autogenerate docs for that as well. Similarly, current concept of unbuildable images would no longer be required because unbuildability would be dictated by lack of support for a required feature.
I'm not sure what you mean by 'feature' here, but I'd say the most common reason an image is unbuildable currently is that the main package required by the image is unavailable, rather than some ancillary package required for a particular feature. Modifying unbuildable images to include unavailable features could be interesting, but would be an extension of current behaviour.
***
Looking forward to your opinions/thoughts.
I think there are some good ideas in here, but I feel that the cost/benefit of the core proposal to replace Jinja with Python doesn't work for me. Specifically, the cost would be high in terms of being a significant rewrite of every image and building additional tooling, plus the operator headaches of switching to the new customisation model. In terms of benefits, it seems they are mostly for contributors rather than users.
I'd like us to explore some of the pain points raised here, and also see if we can determine any others through the kolla klub.
* Can we reduce the image size? Are we installing unnecessary packages? (spoiler, yes) Can we use multi stage builds? * Can we improve testing, to make it clearer what the effects of changing a particular image would be? * What would a 'feature' look like with our current tooling? I'd like to see a concrete example. * What can we learn from the proposed goal [3] to add container images for each project? Could kolla be used in a more distributed manner more amenable to a CI/CD pattern where each project publishes its own images? Could we add a 'python' base distro that is based on the python:3-slim image? I realise the authors of the goal won't like this, but it does add some missing flexibility to their proposal.
[1] https://review.opendev.org/#/c/631647/ [2] https://testinfra.readthedocs.io/en/latest/ [3] https://review.opendev.org/#/c/720107/
***
[1] https://docs.openstack.org/kolla/train/support_matrix.html [2] https://docs.openstack.org/kolla/train/admin/image-building.html
-yoctozepto
-- Mohammed Naser VEXXHOST, Inc.
On Tue, Apr 21, 2020 at 9:12 PM Mohammed Naser <mnaser@vexxhost.com> wrote:
On Tue, Apr 21, 2020 at 5:15 AM Mark Goddard <mark@stackhpc.com> wrote:
On Sat, 18 Apr 2020 at 15:10, Radosław Piliszek <radoslaw.piliszek@gmail.com> wrote:
The above effort could well be coordinated with different projects to reuse bindep contents. So far Kolla does not use bindep because it often installs too much and not enough at the same time. Do note it would still be bindep-less for external services.
I like the idea of using bindep for source images, in combination with multi-stage builds. If we could reduce the length of our package dependency lists by relying on bindep, that would be a nice win. Ideally, between distro package dependencies and bindep, we would need to specify few additional packages.
I think this is a golden ticket of what I'm trying to accomplish and this is what OpenDev is currently doing. Ideally, there _should_ be no distro package dependencies because they should all be modeled inside bindep (using different profiles if needed).
I still think that if we do this, there is a lot of value in being able to do this in-repo because those Dockerfile's will be pretty static. While I appreciate that Kolla has all sorts of neat things to customize images, for a certain operator base, they'd much rather either have a Dockerfile to build off of or an existing base image somewhere.
Kolla could be the thing that drives this, Kolla could become "OpenStack containers", or parts of it could be. However, I think the "customization bits" should ideally stay outside of the scope of this, because we just want to publish an artifact with dependencies installed.
I think we all agree here that bindep is the way to go and whatever we decide on should include "better bindep experience". Could we collect together some concrete ideas how to make the bindep better? (and then suddenly bindep sig appears out of nowhere :O ) I guess Kolla is a great source especially for optional runtime bindep entries, though it can really only offer names of packages per distro rather than insight why they are used (except for git archaelogy of course). We could collaborate with each project to discover real relations between these. Making bindep better would benefit all efforts of deployment (with devstack included) so it's a win-win. For the purpose of this discussion I use the term "bindep" for the bindep.txt but obviously it could mean improving the tool as well. There is a lot to encode: - build time / run time - this is already doable with tags and rather pretty clean, except for no linkage between the two concepts (which is what I tried to attack by "feature" concept) - applicable distro - rather supported - whether optional and when to use (think extras in setup.cfg) - rather supported, wonder if in the cleanest way, but it worked for oslo.cache gate to define functional tests based on backend Any others? -yoctozepto
On Wed, 2020-04-22 at 15:24 +0200, Radosław Piliszek wrote:
On Tue, Apr 21, 2020 at 9:12 PM Mohammed Naser <mnaser@vexxhost.com> wrote:
On Tue, Apr 21, 2020 at 5:15 AM Mark Goddard <mark@stackhpc.com> wrote:
On Sat, 18 Apr 2020 at 15:10, Radosław Piliszek <radoslaw.piliszek@gmail.com> wrote:
The above effort could well be coordinated with different projects to reuse bindep contents. So far Kolla does not use bindep because it often installs too much and not enough at the same time. Do note it would still be bindep-less for external services.
I like the idea of using bindep for source images, in combination with multi-stage builds. If we could reduce the length of our package dependency lists by relying on bindep, that would be a nice win. Ideally, between distro package dependencies and bindep, we would need to specify few additional packages.
I think this is a golden ticket of what I'm trying to accomplish and this is what OpenDev is currently doing. Ideally, there _should_ be no distro package dependencies because they should all be modeled inside bindep (using different profiles if needed).
I still think that if we do this, there is a lot of value in being able to do this in-repo because those Dockerfile's will be pretty static. While I appreciate that Kolla has all sorts of neat things to customize images, for a certain operator base, they'd much rather either have a Dockerfile to build off of or an existing base image somewhere.
Kolla could be the thing that drives this, Kolla could become "OpenStack containers", or parts of it could be. However, I think the "customization bits" should ideally stay outside of the scope of this, because we just want to publish an artifact with dependencies installed.
I think we all agree here that bindep is the way to go and whatever we decide on should include "better bindep experience". Could we collect together some concrete ideas how to make the bindep better? (and then suddenly bindep sig appears out of nowhere :O ) I guess Kolla is a great source especially for optional runtime bindep entries, though it can really only offer names of packages per distro rather than insight why they are used (except for git archaelogy of course). We could collaborate with each project to discover real relations between these. Making bindep better would benefit all efforts of deployment (with devstack included) so it's a win-win.
For the purpose of this discussion I use the term "bindep" for the bindep.txt but obviously it could mean improving the tool as well. There is a lot to encode: - build time / run time - this is already doable with tags and rather pretty clean, except for no linkage between the two concepts (which is what I tried to attack by "feature" concept) - applicable distro - rather supported - whether optional and when to use (think extras in setup.cfg) - rather supported, wonder if in the cleanest way, but it worked for oslo.cache gate to define functional tests based on backend Any others? we have a weird redhat downstream requiremetn that we were debating internally proposing as a change to upstream basicaly in our downstream ci we dont use pip or at least try not too. there are some optional dependices that are needed for optional backend not supproted in our product that do not have package, so those come form pip but otherwise we build an updated version of the distro package and use that.
so the idea our ci folks came up with was listign the proejct itself in bindeps. i found that to be quite strange and was kind fo arguing against that but if we had a convention where each project woudl list the name of the distro package for that project with a "self" tag and we ensureed it is only installed when that tag is used then that would make downstrema ci simpler e.g. the ci job would juse use zull to prepare all the repose and build the rpms then in the job we woudl do "bindep -l newline self | xargs <package mangager> install -y" that would pull in the newly built rpm for that project and use the package depencies in the distor to pull in the set of pacakges we expect. the issue we have today and what having a self tag would solve is we have to maintain a list of dependcies https://github.com/redhat-openstack/octario/blob/master/components_config/16... and then our downstream ci today truncate the requiremets.txt and test-requirements.txt so tehy are empty and run tox with --site-pacakges to use the depency we installed with the package manager. https://github.com/redhat-openstack/octario/blob/master/components_config/16... migi who maintains this part of our down stream ci has already done some experiments to use bindeps instead a moth or two ago wehre he propsoed a downstream patch to make a few small cahnges to the bindeps file to properly include all the deps aso i think this is something that would be useful outside of the container discuss too. i have cc'd migi who can explain this better and maybe separately share his expirence with using bindeps in this way as i have proably explained that terribly :)
-yoctozepto
On Tue, Apr 21, 2020 at 10:59 AM Mark Goddard <mark@stackhpc.com> wrote:
On Sat, 18 Apr 2020 at 15:10, Radosław Piliszek <radoslaw.piliszek@gmail.com> wrote:
My idea is to reuse Kolla engine where it shines: sources collection, system of plugins, hierarchical building; but replace the part that smells - Jinja2 templating. Giving up on Jinja2 might encompass giving up on Dockerfile syntax, but that is optional and depends on what makes it more amenable to avoid further pitfalls.
With giving up on Jinja2, the idea is to generate building recipes fully programmatically from Python.
There would be some nice advantages to this, not least it would be Python rather than Jinja. Also it would be nicely unit testable.
Agreed, my thoughts exactly.
However, Jinja does give us some nice properties. Dockerfiles are relatively WYSIWIG, and this keeps the barrier to contribution low. Often the most that needs to be done to help a contributor get started with kolla is to say 'here is the Dockerfile.j2 for the image you want to change'.
Fair point, however it comes at the cost of maintaibility and deeper analysis as we all agreed.
As Sean rightly pointed out, we have wedded ourselves to Jinja by exposing it as a customisation point, through blocks and overrides. We'd need to replicate that.
Yes, I reread my mail and noticed I really forgot to mention it. I already partially answered about that in reply to Sean. The point is it could actually be easier to replicate than currently. Why? Because we would have programmatically generated names, instead of hoping not to make typos in the templates (which happened). If there was a real problem with some override strategy, at worst we would have to provide a translation layer of sorts. I can't imagine it now though, except for providing completely own Dockerfile.j2, which could just be maintained as legacy mode I guess?
A potential substitute for unit testing could be to generate every supported combination of Dockerfiles, and keep them in the repository. These would need to be updated with each change, but would allow us to see the final content of the images for each distro. Even better than this would be to use TestInfra [2] to validate various expectations about image content (e.g. virtualenv uses py3.6, package foo is installed).
Oh, that is a nice one indeed. Never heard of TestInfra before. This might be a good next step to validate that packages install expected contents.
It would be possible to introduce "Features" - sets of packages to install based on the (distro, arch) tuple. This would result in more flexibility - turning them on/off (some could be optional, some not). There could be more than one optimization strategy regarding when packages get installed: you want only standalone blah-blah? Then Kolla won't be installing XYZ and ABC just because ugma-ugma and tickle-tickle require them and you "could save some space" (TM). In the same vein, features could declare which components are build-time and which are run-time and this would make it straightforward to separate the sides.
This could be neat, although it would be difficult to test and keep working, and would only increase our support matrix. You could do it with Jinja, although it might get a bit unwieldy.
It would be unit-testable, official images would cater for the general usage pattern.
The above effort could well be coordinated with different projects to reuse bindep contents. So far Kolla does not use bindep because it often installs too much and not enough at the same time. Do note it would still be bindep-less for external services.
I like the idea of using bindep for source images, in combination with multi-stage builds. If we could reduce the length of our package dependency lists by relying on bindep, that would be a nice win. Ideally, between distro package dependencies and bindep, we would need to specify few additional packages.
I think we all agree here.
There would still need to exist a general mechanism for providing custom command executions required by some images.
For contributors and cores this new approach would bring more sanity as to the scope of proposed changes. Also, it would be possible to get quick insight into feature support and autogenerate docs for that as well. Similarly, current concept of unbuildable images would no longer be required because unbuildability would be dictated by lack of support for a required feature.
I'm not sure what you mean by 'feature' here, but I'd say the most common reason an image is unbuildable currently is that the main package required by the image is unavailable, rather than some ancillary package required for a particular feature. Modifying unbuildable images to include unavailable features could be interesting, but would be an extension of current behaviour.
A very particular example for the moment would be cyborg-agent, which became unbuildable because OPAE SDK was not available for the supported platforms, while it looks like it is an optional component but there was no way to specify that reasoning (except for a comment but it did not make it it seems). The point is long list of unbuildable images sprinkled with comments is less manageable than specific requirements being unmet by potential targets.
***
Looking forward to your opinions/thoughts.
I think there are some good ideas in here, but I feel that the cost/benefit of the core proposal to replace Jinja with Python doesn't work for me. Specifically, the cost would be high in terms of being a significant rewrite of every image and building additional tooling, plus the operator headaches of switching to the new customisation model. In terms of benefits, it seems they are mostly for contributors rather than users.
It is targeted both at contributors and users as it allows more flexibility on both sides. I agree it is going to be a considerable amount of work but so is anything at this point. Either leave it be and just hope it doesn't make each cycle more painful or innovate.
I'd like us to explore some of the pain points raised here, and also see if we can determine any others through the kolla klub.
This is indeed interesting in terms of Kolla Klub. I really wish to get to know how human users use Kolla. We mostly talk about Kolla Ansible, for obvious reasons, but Kolla is equally important for production.
* Can we reduce the image size? Are we installing unnecessary packages? (spoiler, yes) Can we use multi stage builds?
We can, we are, we want.
* Can we improve testing, to make it clearer what the effects of changing a particular image would be?
TestInfra sounds good.
* What would a 'feature' look like with our current tooling? I'd like to see a concrete example.
See my example above. Similarly for collectd plugins targeted at Intel platforms. This would also document the projects as well, best done with bindep coordination, e.g. I need NFS client in a few containers; NFS client is a feature here.
* What can we learn from the proposed goal [3] to add container images for each project? Could kolla be used in a more distributed manner more amenable to a CI/CD pattern where each project publishes its own images? Could we add a 'python' base distro that is based on the python:3-slim image? I realise the authors of the goal won't like this, but it does add some missing flexibility to their proposal.
bindep is the common denominator, I guess we should push every effort for the sake of better bindep. I wouldn't also mind a switch to "common distro" for the source flavour (likely as an extra for now). -yoctozepto
From: Radosław Piliszek <radoslaw.piliszek@gmail.com> Sent: Wednesday, April 22, 2020 6:11 AM To: Mark Goddard <mark@stackhpc.com> Cc: openstack-discuss <openstack-discuss@lists.openstack.org> Subject: Re: [all] [kolla] Kolla Builder - next generation [...] A very particular example for the moment would be cyborg-agent, which became unbuildable because OPAE SDK was not available for the supported platforms, while it looks like it is an optional component but there was no way to specify that reasoning (except for a comment but it did not make it it seems).
We are aware of this issue. A proposal to remove this OPAE dependency has been in the works. We will bring this up in the Victoria PTG and aim to make it a goal for the V release. Regards, Sundar
On Thu, 28 May 2020 at 05:36, Nadathur, Sundar <sundar.nadathur@intel.com> wrote:
From: Radosław Piliszek <radoslaw.piliszek@gmail.com> Sent: Wednesday, April 22, 2020 6:11 AM To: Mark Goddard <mark@stackhpc.com> Cc: openstack-discuss <openstack-discuss@lists.openstack.org> Subject: Re: [all] [kolla] Kolla Builder - next generation [...] A very particular example for the moment would be cyborg-agent, which became unbuildable because OPAE SDK was not available for the supported platforms, while it looks like it is an optional component but there was no way to specify that reasoning (except for a comment but it did not make it it seems).
We are aware of this issue. A proposal to remove this OPAE dependency has been in the works. We will bring this up in the Victoria PTG and aim to make it a goal for the V release.
Hi Sundar, thanks for the reply. Can you confirm whether this is a hard dependency on OPAE? If so, we will be unable to support Cyborg on CentOS in Ussuri.
Regards, Sundar
participants (5)
-
Mark Goddard
-
Mohammed Naser
-
Nadathur, Sundar
-
Radosław Piliszek
-
Sean Mooney