[kolla] Plan to deprecate binary and unify on single distrubition
Hello Koalas,
On the PTG we have discussed two topics:
1) Deprecate and drop binary type of Kolla images 2) Use a common base (single Linux distribution) for Kolla images
This is a call for feedback - for people that have not been attending the PTG.
What this essentially mean for consumers:
1) In Yoga cycle we will deprecate binary type of Kolla images, and in Z cycle those will be dropped. 2) We are not going to support CentOS Stream 9 (cs9) as a base operating system, and the source type build will rely on CentOS Stream 8 in Z release. 3) Beginning from A release Kolla will build only Debian source images - but Kolla-Ansible will still support deployment of those images on CentOS/Ubuntu/Debian Host operating systems (and Rocky Linux to be added in Yoga to that mix).
Justification: The Kolla project team is limited in numbers, therefore supporting current broad mix of operating systems (especially with CentOS Stream 9 ,,on the way’’) is a significant maintenance burden. By dropping binary type of images - users would be running more tested images (since Kolla/Kolla-Ansible CI runs source images jobs as voting). In Xena we’ve already changed the default image type Kolla-Ansible uses to source. We also feel that using a unified base OS for Kolla container images is a way to remove some of the maintenance burden (including CI cycles and
Request for feedback: If any of those changes is a no go from your perspective - we’d like to hear your opinions.
Best regards, Michal Nasiadka
On Thu, 2021-11-18 at 13:33 +0100, Michał Nasiadka wrote:
Hello Koalas,
On the PTG we have discussed two topics:
- Deprecate and drop binary type of Kolla images
- Use a common base (single Linux distribution) for Kolla images
This is a call for feedback - for people that have not been attending the PTG.
What this essentially mean for consumers:
- In Yoga cycle we will deprecate binary type of Kolla images, and in Z cycle those will be dropped.
- We are not going to support CentOS Stream 9 (cs9) as a base operating system, and the source type build will rely on CentOS Stream 8 in Z release.
- Beginning from A release Kolla will build only Debian source images - but Kolla-Ansible will still support deployment of those images on CentOS/Ubuntu/Debian Host operating systems (and Rocky Linux to be added in Yoga to that mix).
Justification: The Kolla project team is limited in numbers, therefore supporting current broad mix of operating systems (especially with CentOS Stream 9 ,,on the way’’) is a significant maintenance burden. By dropping binary type of images - users would be running more tested images (since Kolla/Kolla-Ansible CI runs source images jobs as voting). In Xena we’ve already changed the default image type Kolla-Ansible uses to source. We also feel that using a unified base OS for Kolla container images is a way to remove some of the maintenance burden (including CI cycles and
Request for feedback: If any of those changes is a no go from your perspective - we’d like to hear your opinions.
i only have reason to use kolla-ansible on small personal clouds at home so either way this will have limited direct affect on me but just wanted to give some toughts: kolla has been my favorit way to deploy openstack for a very long time, one of the reason i liked it was the fact that it was distro independent, simple ot use/configure and it support source and binary installs. i have almost always defautl to ubuntu source although in the rare case si used centos i always used centos binary.
i almost never used binary installs on debian based disto and on rpm based distos never really used soruce, im not sure if im the only one that did that. its not because i trusted the rpm package more by the way it just seam to be waht was tested more when i cahged to kolla devs on irc so i avoid ubuntu bindary and centos source as a result.
with that in mind debian source is not really contoversial to me, i had one quetion on that however. will the support for the other distos be kept in the kolla images but not extended or will it be dropped. i asume the plan is to remvoe the templating for other distos in A based on point 3 above.
the only other thing i wanted to point out is that while i have had some succes gettign ubuntu soruce image to run on centos host in the past it will be tricky if kolla every want to supprot selinx/apparmor. that was the main barrier i faced but there can be other. speficialy ovs and libvirt can be somewhat picky about the kernel on which they run. most of the openstack service will likely not care that the contaienr os does not match the host but some of the "system" depenciy like libvirt/ovs might. a way to address taht would be to supprot using external images for those servie form dockerhub/quay e.g. use the offical upstream mariadb image or libvirt or rabbit if that is infact a porblem.
anyway its totally understandable that if you do not have contirbutor that are able to support the other distors that you would remove the supprot. espicaly with the move away form using kolla image in ooo of late and presumable a reduction in redhat contibutions to keep centos supprot alive.
anyway the chagne while sad to see just form a proejct health point of vew are sad would not be enough to prevent me personally form using or recommendign kolla and kolla-ansibel. if you had prorpsed the opistte of centos binary only that would be much more concerning to me. there are some advantages to source installs that you are preservging in this change that i am glad will not be lost.
one last tought is that if only one disto will be supproted for building images in the future with only soruce installs supproted, it might be worth considering if alpine or the alpine/debian-lite python 3 iamge should be revaluated as the base for an even lighter set of contianers rather then the base os image. i am kind fo assumeing that at some poitn the non python noopenstack contaienr would be used form the offial images at some point rahter then kolla contining to maintain them with the above suggestion. So this may not be applicable now and really would likely be an A or post A thing.
Best regards, Michal Nasiadka
Hi Sean,
On 18 Nov 2021, at 16:04, Sean Mooney smooney@redhat.com wrote:
On Thu, 2021-11-18 at 13:33 +0100, Michał Nasiadka wrote:
Hello Koalas,
On the PTG we have discussed two topics:
- Deprecate and drop binary type of Kolla images
- Use a common base (single Linux distribution) for Kolla images
This is a call for feedback - for people that have not been attending the PTG.
What this essentially mean for consumers:
- In Yoga cycle we will deprecate binary type of Kolla images, and in Z cycle those will be dropped.
- We are not going to support CentOS Stream 9 (cs9) as a base operating system, and the source type build will rely on CentOS Stream 8 in Z release.
- Beginning from A release Kolla will build only Debian source images - but Kolla-Ansible will still support deployment of those images on CentOS/Ubuntu/Debian Host operating systems (and Rocky Linux to be added in Yoga to that mix).
Justification: The Kolla project team is limited in numbers, therefore supporting current broad mix of operating systems (especially with CentOS Stream 9 ,,on the way’’) is a significant maintenance burden. By dropping binary type of images - users would be running more tested images (since Kolla/Kolla-Ansible CI runs source images jobs as voting). In Xena we’ve already changed the default image type Kolla-Ansible uses to source. We also feel that using a unified base OS for Kolla container images is a way to remove some of the maintenance burden (including CI cycles and
Request for feedback: If any of those changes is a no go from your perspective - we’d like to hear your opinions.
i only have reason to use kolla-ansible on small personal clouds at home so either way this will have limited direct affect on me but just wanted to give some toughts: kolla has been my favorit way to deploy openstack for a very long time, one of the reason i liked it was the fact that it was distro independent, simple ot use/configure and it support source and binary installs. i have almost always defautl to ubuntu source although in the rare case si used centos i always used centos binary.
i almost never used binary installs on debian based disto and on rpm based distos never really used soruce, im not sure if im the only one that did that. its not because i trusted the rpm package more by the way it just seam to be waht was tested more when i cahged to kolla devs on irc so i avoid ubuntu bindary and centos source as a result.
with that in mind debian source is not really contoversial to me, i had one quetion on that however. will the support for the other distos be kept in the kolla images but not extended or will it be dropped. i asume the plan is to remvoe the templating for other distos in A based on point 3 above.
Yes, currently that’s the plan - to remove templating for other distros and entries for them in kolla-build code.
the only other thing i wanted to point out is that while i have had some succes gettign ubuntu soruce image to run on centos host in the past it will be tricky if kolla every want to supprot selinx/apparmor. that was the main barrier i faced but there can be other. speficialy ovs and libvirt can be somewhat picky about the kernel on which they run. most of the openstack service will likely not care that the contaienr os does not match the host but some of the "system" depenciy like libvirt/ovs might. a way to address taht would be to supprot using external images for those servie form dockerhub/quay e.g. use the offical upstream mariadb image or libvirt or rabbit if that is infact a porblem.
We’ve been discussing about moving OVS and Libvirt deployment to be deployed on OS level (not in containers) - if that will be required.
anyway its totally understandable that if you do not have contirbutor that are able to support the other distors that you would remove the supprot. espicaly with the move away form using kolla image in ooo of late and presumable a reduction in redhat contibutions to keep centos supprot alive.
anyway the chagne while sad to see just form a proejct health point of vew are sad would not be enough to prevent me personally form using or recommendign kolla and kolla-ansibel. if you had prorpsed the opistte of centos binary only that would be much more concerning to me. there are some advantages to source installs that you are preservging in this change that i am glad will not be lost.
That’s good to see.
one last tought is that if only one disto will be supproted for building images in the future with only soruce installs supproted, it might be worth considering if alpine or the alpine/debian-lite python 3 iamge should be revaluated as the base for an even lighter set of contianers rather then the base os image. i am kind fo assumeing that at some poitn the non python noopenstack contaienr would be used form the offial images at some point rahter then kolla contining to maintain them with the above suggestion. So this may not be applicable now and really would likely be an A or post A thing.
That was in the PTG notes, but I don’t think we have considered a separate base OS for python-based Kolla images (and a separate for those that need it or use official Docker Hub/Quay.io http://quay.io/ images for those services).
Thanks for your input!
Best regards, Michal Nasiadka
W dniu 18.11.2021 o 16:04, Sean Mooney pisze:
one last tought is that if only one disto will be supproted for building images in the future with only soruce installs supproted, it might be worth considering if alpine or the alpine/debian-lite python 3 iamge should be revaluated as the base for an even lighter set of contianers rather then the base os image.
Using one distribution which covers all images makes life simple in Kolla.
Alpine may not cover all our needs as some software still requires things present in glibc, missing in musl libc.
And while openstack-base image probably could be built using one of python lite containers there are images derived from it which install additional distro packages. So instead of taking packages from one distribution (Debian) we would take from Debian or Alpine. Which could get out of sync too easily.
On 18/11/21 8:33 pm, Michał Nasiadka wrote:
- Deprecate and drop binary type of Kolla images
My globals.yaml has #kolla_install_type: "binary"
So will this mean it needs to switch to 'source' from Yoga onward, and the containers will have to be built some how? Can you point me to something to read on that?
- Use a common base (single Linux distribution) for Kolla images
I'm not experienced enough with Kolla to know whether this will affect me, so I will roll with it and figure it out as we go.
Thanks, Greg.
Hi Gregory,
On 19 Nov 2021, at 01:33, Gregory Orange gregory.orange@pawsey.org.au wrote:
On 18/11/21 8:33 pm, Michał Nasiadka wrote:
- Deprecate and drop binary type of Kolla images
My globals.yaml has #kolla_install_type: "binary"
So will this mean it needs to switch to 'source' from Yoga onward, and the containers will have to be built some how? Can you point me to something to read on that?
If that’s commented out, that means with Xena you will get source images deployed (during an upgrade or on a fresh install).
- Use a common base (single Linux distribution) for Kolla images
I'm not experienced enough with Kolla to know whether this will affect me, so I will roll with it and figure it out as we go.
Thanks, Greg.
Thanks, Michal
On 19/11/21 4:19 pm, Michał Nasiadka wrote:
My globals.yaml has #kolla_install_type: "binary"
So will this mean it needs to switch to 'source' from Yoga onward, and the containers will have to be built some how? Can you point me to something to read on that?
If that’s commented out, that means with Xena you will get source images deployed (during an upgrade or on a fresh install).
So am I going to need to familiarise myself with some build process, such as https://docs.openstack.org/kolla/train/admin/image-building.html ?
On Fri, 2021-11-19 at 19:27 +0800, Gregory Orange wrote:
On 19/11/21 4:19 pm, Michał Nasiadka wrote:
My globals.yaml has #kolla_install_type: "binary"
So will this mean it needs to switch to 'source' from Yoga onward, and the containers will have to be built some how? Can you point me to something to read on that?
If that’s commented out, that means with Xena you will get source images deployed (during an upgrade or on a fresh install).
So am I going to need to familiarise myself with some build process, such as https://docs.openstack.org/kolla/train/admin/image-building.html ?
no the soruce images are also published to docker/quay registryso if you are not building them today and pulling them from there it will work the same if you are building image today there is no really different form a commandline point of view for source vs binary. the main different is the soruce images install all the deps from pip and the servcis form the offical tarballs into a python virtual env within the contaianer image. so unless you have explcitly set kolla_install_type: binay or the distroy you should not need to change anything hopefuly in your gobal.yaml ideally.
Interesting.
This is probably a little bit off-topic but I find it very interesting that a majority of all the big OpenStack clouds out there is running on containers based and a lot of them using “LOKI” that was talked about so much in the OpenInfra Live Keynotes.
What I don’t understand is that, with all these limited resources, there is no joint effort in the OpenStack ecosystem to solve the container deliverables issue and then just have all deployments/tooling use the same.
Maybe that what they are doing though, using Kolla images… but then, wouldn’t they contribute more and the below not be a problem *makes me wonder*
Sorry for off-topic loud thinking.
Best regards Tobias
On 18 Nov 2021, at 13:33, Michał Nasiadka mnasiadka@gmail.com wrote:
Hello Koalas,
On the PTG we have discussed two topics:
- Deprecate and drop binary type of Kolla images
- Use a common base (single Linux distribution) for Kolla images
This is a call for feedback - for people that have not been attending the PTG.
What this essentially mean for consumers:
- In Yoga cycle we will deprecate binary type of Kolla images, and in Z cycle those will be dropped.
- We are not going to support CentOS Stream 9 (cs9) as a base operating system, and the source type build will rely on CentOS Stream 8 in Z release.
- Beginning from A release Kolla will build only Debian source images - but Kolla-Ansible will still support deployment of those images on CentOS/Ubuntu/Debian Host operating systems (and Rocky Linux to be added in Yoga to that mix).
Justification: The Kolla project team is limited in numbers, therefore supporting current broad mix of operating systems (especially with CentOS Stream 9 ,,on the way’’) is a significant maintenance burden. By dropping binary type of images - users would be running more tested images (since Kolla/Kolla-Ansible CI runs source images jobs as voting). In Xena we’ve already changed the default image type Kolla-Ansible uses to source. We also feel that using a unified base OS for Kolla container images is a way to remove some of the maintenance burden (including CI cycles and
Request for feedback: If any of those changes is a no go from your perspective - we’d like to hear your opinions.
Best regards, Michal Nasiadka
For OpenStack, I've been using centos-binary containers on CentOS. It's been working very well.
I don't have much concern on supporting single-distro based container on different distro hosts. That's actually one of the beauties of container, which self-contains all packages and dependencies. We used to have a product based on single-distro container and support different distro hosts. Just need to be careful when container needs any resource from the host, like kernel, mounted host filesystem, networking, privilege, etc.
Regarding to source container, what's the purpose of it? Is it allow user to get the package based on some specific source that is not provided by any existing package repo? In that case, I'd assume user should always build their own source container. Then what's the purpose of providing pre-build source container? Also, are those non-core containers, like MariaDB, HAProxy, RabbitMQ, etc. all built from source code?
Thanks! Tony ________________________________________ From: Michał Nasiadka mnasiadka@gmail.com Sent: November 18, 2021 04:33 AM To: openstack-discuss Subject: [kolla] Plan to deprecate binary and unify on single distrubition
Hello Koalas,
On the PTG we have discussed two topics:
1) Deprecate and drop binary type of Kolla images 2) Use a common base (single Linux distribution) for Kolla images
This is a call for feedback - for people that have not been attending the PTG.
What this essentially mean for consumers:
1) In Yoga cycle we will deprecate binary type of Kolla images, and in Z cycle those will be dropped. 2) We are not going to support CentOS Stream 9 (cs9) as a base operating system, and the source type build will rely on CentOS Stream 8 in Z release. 3) Beginning from A release Kolla will build only Debian source images - but Kolla-Ansible will still support deployment of those images on CentOS/Ubuntu/Debian Host operating systems (and Rocky Linux to be added in Yoga to that mix).
Justification: The Kolla project team is limited in numbers, therefore supporting current broad mix of operating systems (especially with CentOS Stream 9 ,,on the way’’) is a significant maintenance burden. By dropping binary type of images - users would be running more tested images (since Kolla/Kolla-Ansible CI runs source images jobs as voting). In Xena we’ve already changed the default image type Kolla-Ansible uses to source. We also feel that using a unified base OS for Kolla container images is a way to remove some of the maintenance burden (including CI cycles and
Request for feedback: If any of those changes is a no go from your perspective - we’d like to hear your opinions.
Best regards, Michal Nasiadka
The user doesn't have to build the images that are source based. Ready built source-based images are available: https://quay.io/search?q=kolla
Binary-based Kolla docker images are installing (at image build time) OpenStack projects via system packages, while source based images are symply adding projects' Python source code.
Just as you said, the user shouldn't care much about distro inside images. But makes things a lot easier for developers to maintain less images and without working around different versions and dependencies provided by the system packages of various distros.
Regards, Adrian Andreias
On Wed, Dec 1, 2021 at 10:31 PM Tony Liu tonyliu0592@hotmail.com wrote:
For OpenStack, I've been using centos-binary containers on CentOS. It's been working very well.
I don't have much concern on supporting single-distro based container on different distro hosts. That's actually one of the beauties of container, which self-contains all packages and dependencies. We used to have a product based on single-distro container and support different distro hosts. Just need to be careful when container needs any resource from the host, like kernel, mounted host filesystem, networking, privilege, etc.
Regarding to source container, what's the purpose of it? Is it allow user to get the package based on some specific source that is not provided by any existing package repo? In that case, I'd assume user should always build their own source container. Then what's the purpose of providing pre-build source container? Also, are those non-core containers, like MariaDB, HAProxy, RabbitMQ, etc. all built from source code?
Thanks! Tony ________________________________________ From: Michał Nasiadka mnasiadka@gmail.com Sent: November 18, 2021 04:33 AM To: openstack-discuss Subject: [kolla] Plan to deprecate binary and unify on single distrubition
Hello Koalas,
On the PTG we have discussed two topics:
- Deprecate and drop binary type of Kolla images
- Use a common base (single Linux distribution) for Kolla images
This is a call for feedback - for people that have not been attending the PTG.
What this essentially mean for consumers:
- In Yoga cycle we will deprecate binary type of Kolla images, and in Z
cycle those will be dropped. 2) We are not going to support CentOS Stream 9 (cs9) as a base operating system, and the source type build will rely on CentOS Stream 8 in Z release. 3) Beginning from A release Kolla will build only Debian source images - but Kolla-Ansible will still support deployment of those images on CentOS/Ubuntu/Debian Host operating systems (and Rocky Linux to be added in Yoga to that mix).
Justification: The Kolla project team is limited in numbers, therefore supporting current broad mix of operating systems (especially with CentOS Stream 9 ,,on the way’’) is a significant maintenance burden. By dropping binary type of images - users would be running more tested images (since Kolla/Kolla-Ansible CI runs source images jobs as voting). In Xena we’ve already changed the default image type Kolla-Ansible uses to source. We also feel that using a unified base OS for Kolla container images is a way to remove some of the maintenance burden (including CI cycles and
Request for feedback: If any of those changes is a no go from your perspective - we’d like to hear your opinions.
Best regards, Michal Nasiadka
Since I've never used source container and not able to find answers from doc, bear my dummy questions. BTW, I am not able to figure it out from Dockerfile.j2.
Take centos-source-mariadb as an example, is it built by 1) installing packages from package repo or 2) download source code, compile and build packages and install self-build packages? In case of 2), how can I tell where and which release source code is downloaded and how it's compiled and built? In case of 1), what's the difference here from centos-binary-mariadb?
Similarly, take centos-source-keystone as another example, is it built by 1) installing package from eg. centos/8-stream/cloud/x86_64/openstack-ussuri/ or 2) download source code from Github, build packages locally and install them? The same questions here, for 1) what's the difference from binary container, for 2) how can I tell which revision the source code is downloaded?
Or, the source container is binary container with source code?
Thanks! Tony ________________________________________ Fro: Adrian Andreias adrian@fleio.com Sent: December 1, 2021 01:30 PM To: Tony Liu Cc: Michał Nasiadka; openstack-discuss Subject: Re: [kolla] Plan to deprecate binary and unify on single distrubition
The user doesn't have to build the images that are source based. Ready built source-based images are available: https://quay.io/search?q=kolla
Binary-based Kolla docker images are installing (at image build time) OpenStack projects via system packages, while source based images are symply adding projects' Python source code.
Just as you said, the user shouldn't care much about distro inside images. But makes things a lot easier for developers to maintain less images and without working around different versions and dependencies provided by the system packages of various distros.
Regards, Adrian Andreias
On Wed, Dec 1, 2021 at 10:31 PM Tony Liu <tonyliu0592@hotmail.commailto:tonyliu0592@hotmail.com> wrote: For OpenStack, I've been using centos-binary containers on CentOS. It's been working very well.
I don't have much concern on supporting single-distro based container on different distro hosts. That's actually one of the beauties of container, which self-contains all packages and dependencies. We used to have a product based on single-distro container and support different distro hosts. Just need to be careful when container needs any resource from the host, like kernel, mounted host filesystem, networking, privilege, etc.
Regarding to source container, what's the purpose of it? Is it allow user to get the package based on some specific source that is not provided by any existing package repo? In that case, I'd assume user should always build their own source container. Then what's the purpose of providing pre-build source container? Also, are those non-core containers, like MariaDB, HAProxy, RabbitMQ, etc. all built from source code?
Thanks! Tony ________________________________________ From: Michał Nasiadka <mnasiadka@gmail.commailto:mnasiadka@gmail.com> Sent: November 18, 2021 04:33 AM To: openstack-discuss Subject: [kolla] Plan to deprecate binary and unify on single distrubition
Hello Koalas,
On the PTG we have discussed two topics:
1) Deprecate and drop binary type of Kolla images 2) Use a common base (single Linux distribution) for Kolla images
This is a call for feedback - for people that have not been attending the PTG.
What this essentially mean for consumers:
1) In Yoga cycle we will deprecate binary type of Kolla images, and in Z cycle those will be dropped. 2) We are not going to support CentOS Stream 9 (cs9) as a base operating system, and the source type build will rely on CentOS Stream 8 in Z release. 3) Beginning from A release Kolla will build only Debian source images - but Kolla-Ansible will still support deployment of those images on CentOS/Ubuntu/Debian Host operating systems (and Rocky Linux to be added in Yoga to that mix).
Justification: The Kolla project team is limited in numbers, therefore supporting current broad mix of operating systems (especially with CentOS Stream 9 ,,on the way’’) is a significant maintenance burden. By dropping binary type of images - users would be running more tested images (since Kolla/Kolla-Ansible CI runs source images jobs as voting). In Xena we’ve already changed the default image type Kolla-Ansible uses to source. We also feel that using a unified base OS for Kolla container images is a way to remove some of the maintenance burden (including CI cycles and
Request for feedback: If any of those changes is a no go from your perspective - we’d like to hear your opinions.
Best regards, Michal Nasiadka
"source" refers here to OpenStack projects source code: https://opendev.org/openstack/kolla/src/commit/1a9d5a1a42c86b447af371ab5e086...
There is no reason to build MariaDB from source. System package are always used (regardless of binary/source config): https://opendev.org/openstack/kolla/src/commit/1a9d5a1a42c86b447af371ab5e086... https://opendev.org/openstack/kolla/src/commit/1a9d5a1a42c86b447af371ab5e086...
Image names like "centos-source-mariadb" are probably just kept for consistency.
If you want to experiment and see what's happening, you can build images yourself: https://docs.openstack.org/kolla/latest/admin/image-building.html
Anyhow, the thread is about any blockers on Michał's proposal. I think it makes sense: less ground to cover and effort will be focused on a few reliable images. The only possible issue I can imagine would be a project that already has massive image customizations that are based on binary images. Otherwise, minor customizations or standard images should not impact change from binary to source nor to single distro.
Regards, Adrian Andreias
On Thu, Dec 2, 2021 at 12:04 AM Tony Liu tonyliu0592@hotmail.com wrote:
Since I've never used source container and not able to find answers from doc, bear my dummy questions. BTW, I am not able to figure it out from Dockerfile.j2.
Take centos-source-mariadb as an example, is it built by 1) installing packages from package repo or 2) download source code, compile and build packages and install self-build packages? In case of 2), how can I tell where and which release source code is downloaded and how it's compiled and built? In case of 1), what's the difference here from centos-binary-mariadb?
Similarly, take centos-source-keystone as another example, is it built by
- installing package from eg.
centos/8-stream/cloud/x86_64/openstack-ussuri/ or 2) download source code from Github, build packages locally and install them? The same questions here, for 1) what's the difference from binary container, for 2) how can I tell which revision the source code is downloaded?
Or, the source container is binary container with source code?
Thanks! Tony ________________________________________ Fro: Adrian Andreias adrian@fleio.com Sent: December 1, 2021 01:30 PM To: Tony Liu Cc: Michał Nasiadka; openstack-discuss Subject: Re: [kolla] Plan to deprecate binary and unify on single distrubition
The user doesn't have to build the images that are source based. Ready built source-based images are available: https://quay.io/search?q=kolla
Binary-based Kolla docker images are installing (at image build time) OpenStack projects via system packages, while source based images are symply adding projects' Python source code.
Just as you said, the user shouldn't care much about distro inside images. But makes things a lot easier for developers to maintain less images and without working around different versions and dependencies provided by the system packages of various distros.
Regards, Adrian Andreias
On Wed, Dec 1, 2021 at 10:31 PM Tony Liu <tonyliu0592@hotmail.commailto: tonyliu0592@hotmail.com> wrote: For OpenStack, I've been using centos-binary containers on CentOS. It's been working very well.
I don't have much concern on supporting single-distro based container on different distro hosts. That's actually one of the beauties of container, which self-contains all packages and dependencies. We used to have a product based on single-distro container and support different distro hosts. Just need to be careful when container needs any resource from the host, like kernel, mounted host filesystem, networking, privilege, etc.
Regarding to source container, what's the purpose of it? Is it allow user to get the package based on some specific source that is not provided by any existing package repo? In that case, I'd assume user should always build their own source container. Then what's the purpose of providing pre-build source container? Also, are those non-core containers, like MariaDB, HAProxy, RabbitMQ, etc. all built from source code?
Thanks! Tony ________________________________________ From: Michał Nasiadka <mnasiadka@gmail.commailto:mnasiadka@gmail.com> Sent: November 18, 2021 04:33 AM To: openstack-discuss Subject: [kolla] Plan to deprecate binary and unify on single distrubition
Hello Koalas,
On the PTG we have discussed two topics:
- Deprecate and drop binary type of Kolla images
- Use a common base (single Linux distribution) for Kolla images
This is a call for feedback - for people that have not been attending the PTG.
What this essentially mean for consumers:
- In Yoga cycle we will deprecate binary type of Kolla images, and in Z
cycle those will be dropped. 2) We are not going to support CentOS Stream 9 (cs9) as a base operating system, and the source type build will rely on CentOS Stream 8 in Z release. 3) Beginning from A release Kolla will build only Debian source images - but Kolla-Ansible will still support deployment of those images on CentOS/Ubuntu/Debian Host operating systems (and Rocky Linux to be added in Yoga to that mix).
Justification: The Kolla project team is limited in numbers, therefore supporting current broad mix of operating systems (especially with CentOS Stream 9 ,,on the way’’) is a significant maintenance burden. By dropping binary type of images - users would be running more tested images (since Kolla/Kolla-Ansible CI runs source images jobs as voting). In Xena we’ve already changed the default image type Kolla-Ansible uses to source. We also feel that using a unified base OS for Kolla container images is a way to remove some of the maintenance burden (including CI cycles and
Request for feedback: If any of those changes is a no go from your perspective - we’d like to hear your opinions.
Best regards, Michal Nasiadka
Thank you Adrian for clarifications! I am good with Michał's proposal, although it will add a bit overhead to rpm-operator when need to look into container. For example, you need to keep it in mind that the container is Debian based and run dpkg instead of rpm.
Tony ________________________________________ From: Adrian Andreias adrian@fleio.com Sent: December 1, 2021 03:04 PM To: Tony Liu Cc: Michał Nasiadka; openstack-discuss Subject: Re: [kolla] Plan to deprecate binary and unify on single distrubition
"source" refers here to OpenStack projects source code: https://opendev.org/openstack/kolla/src/commit/1a9d5a1a42c86b447af371ab5e086...
There is no reason to build MariaDB from source. System package are always used (regardless of binary/source config): https://opendev.org/openstack/kolla/src/commit/1a9d5a1a42c86b447af371ab5e086... https://opendev.org/openstack/kolla/src/commit/1a9d5a1a42c86b447af371ab5e086...
Image names like "centos-source-mariadb" are probably just kept for consistency.
If you want to experiment and see what's happening, you can build images yourself: https://docs.openstack.org/kolla/latest/admin/image-building.html
Anyhow, the thread is about any blockers on Michał's proposal. I think it makes sense: less ground to cover and effort will be focused on a few reliable images. The only possible issue I can imagine would be a project that already has massive image customizations that are based on binary images. Otherwise, minor customizations or standard images should not impact change from binary to source nor to single distro.
Regards, Adrian Andreias
On Thu, Dec 2, 2021 at 12:04 AM Tony Liu <tonyliu0592@hotmail.commailto:tonyliu0592@hotmail.com> wrote: Since I've never used source container and not able to find answers from doc, bear my dummy questions. BTW, I am not able to figure it out from Dockerfile.j2.
Take centos-source-mariadb as an example, is it built by 1) installing packages from package repo or 2) download source code, compile and build packages and install self-build packages? In case of 2), how can I tell where and which release source code is downloaded and how it's compiled and built? In case of 1), what's the difference here from centos-binary-mariadb?
Similarly, take centos-source-keystone as another example, is it built by 1) installing package from eg. centos/8-stream/cloud/x86_64/openstack-ussuri/ or 2) download source code from Github, build packages locally and install them? The same questions here, for 1) what's the difference from binary container, for 2) how can I tell which revision the source code is downloaded?
Or, the source container is binary container with source code?
Thanks! Tony ________________________________________ Fro: Adrian Andreias <adrian@fleio.commailto:adrian@fleio.com> Sent: December 1, 2021 01:30 PM To: Tony Liu Cc: Michał Nasiadka; openstack-discuss Subject: Re: [kolla] Plan to deprecate binary and unify on single distrubition
The user doesn't have to build the images that are source based. Ready built source-based images are available: https://quay.io/search?q=kolla
Binary-based Kolla docker images are installing (at image build time) OpenStack projects via system packages, while source based images are symply adding projects' Python source code.
Just as you said, the user shouldn't care much about distro inside images. But makes things a lot easier for developers to maintain less images and without working around different versions and dependencies provided by the system packages of various distros.
Regards, Adrian Andreias
On Wed, Dec 1, 2021 at 10:31 PM Tony Liu <tonyliu0592@hotmail.commailto:tonyliu0592@hotmail.com<mailto:tonyliu0592@hotmail.commailto:tonyliu0592@hotmail.com>> wrote: For OpenStack, I've been using centos-binary containers on CentOS. It's been working very well.
I don't have much concern on supporting single-distro based container on different distro hosts. That's actually one of the beauties of container, which self-contains all packages and dependencies. We used to have a product based on single-distro container and support different distro hosts. Just need to be careful when container needs any resource from the host, like kernel, mounted host filesystem, networking, privilege, etc.
Regarding to source container, what's the purpose of it? Is it allow user to get the package based on some specific source that is not provided by any existing package repo? In that case, I'd assume user should always build their own source container. Then what's the purpose of providing pre-build source container? Also, are those non-core containers, like MariaDB, HAProxy, RabbitMQ, etc. all built from source code?
Thanks! Tony ________________________________________ From: Michał Nasiadka <mnasiadka@gmail.commailto:mnasiadka@gmail.com<mailto:mnasiadka@gmail.commailto:mnasiadka@gmail.com>> Sent: November 18, 2021 04:33 AM To: openstack-discuss Subject: [kolla] Plan to deprecate binary and unify on single distrubition
Hello Koalas,
On the PTG we have discussed two topics:
1) Deprecate and drop binary type of Kolla images 2) Use a common base (single Linux distribution) for Kolla images
This is a call for feedback - for people that have not been attending the PTG.
What this essentially mean for consumers:
1) In Yoga cycle we will deprecate binary type of Kolla images, and in Z cycle those will be dropped. 2) We are not going to support CentOS Stream 9 (cs9) as a base operating system, and the source type build will rely on CentOS Stream 8 in Z release. 3) Beginning from A release Kolla will build only Debian source images - but Kolla-Ansible will still support deployment of those images on CentOS/Ubuntu/Debian Host operating systems (and Rocky Linux to be added in Yoga to that mix).
Justification: The Kolla project team is limited in numbers, therefore supporting current broad mix of operating systems (especially with CentOS Stream 9 ,,on the way’’) is a significant maintenance burden. By dropping binary type of images - users would be running more tested images (since Kolla/Kolla-Ansible CI runs source images jobs as voting). In Xena we’ve already changed the default image type Kolla-Ansible uses to source. We also feel that using a unified base OS for Kolla container images is a way to remove some of the maintenance burden (including CI cycles and
Request for feedback: If any of those changes is a no go from your perspective - we’d like to hear your opinions.
Best regards, Michal Nasiadka
On 11/18/21 1:33 PM, Michał Nasiadka wrote:
Hello Koalas,
On the PTG we have discussed two topics:
- Deprecate and drop binary type of Kolla images
- Use a common base (single Linux distribution) for Kolla images
This is a call for feedback - for people that have not been attending the PTG.
What this essentially mean for consumers:
- In Yoga cycle we will deprecate binary type of Kolla images, and in Z cycle those will be dropped.
- We are not going to support CentOS Stream 9 (cs9) as a base operating system, and the source type build will rely on CentOS Stream 8 in Z release.
- Beginning from A release Kolla will build only Debian source images - but Kolla-Ansible will still support deployment of those images on CentOS/Ubuntu/Debian Host operating systems (and Rocky Linux to be added in Yoga to that mix).
Justification: The Kolla project team is limited in numbers, therefore supporting current broad mix of operating systems (especially with CentOS Stream 9 ,,on the way’’) is a significant maintenance burden. By dropping binary type of images - users would be running more tested images (since Kolla/Kolla-Ansible CI runs source images jobs as voting). In Xena we’ve already changed the default image type Kolla-Ansible uses to source. We also feel that using a unified base OS for Kolla container images is a way to remove some of the maintenance burden (including CI cycles and
Request for feedback: If any of those changes is a no go from your perspective - we’d like to hear your opinions.
Best regards, Michal Nasiadka
Hi,
While I fully support moving to a Debian only world (anyone who knows me isn't surprised to read this...), I really wonder if moving to a "source only" world is the right step.
If you don't have enough people, why don't you just move to a Debian package only type of install, rather than re-implementing all by yourself? The very point of packages is to abstract for you many things, like for example, the fact that Nova has a reserved UID/GID of 64060 in Debian, and will always use that one, and automatically set that up for you, plus put the Nova user in the right group... Or if you need another example, making sure that swift-proxy starts with the right "route = .* addheader:Date: ${httptime[]}" uwsgi parameter... etc.
Now, if you have a limited amount of contributors, all these tiny tweaks that the package maintainers are doing for you, you'll have to re-implement them yourself. Worse: it's going to be even more complicated if you put stuff in containers.
Using source only images, you also don't get all the testing that's done by distro on funny arch which you probably didn't think of. Yes, the openvswitch package in Debian is tested and working on the 10 official Debian arch. The lz4tools contains a 32 bits patch. Or is it that you don't care about anything else than x86_64?
So why don't you just take the other direction, and drop the source images? I'm sure this will save you a lot of work. Plus myself and Michal will make sure everything works, and is available in time for every release, like we always did (did you notice Debian is often (if not always) the first distro to have available packages for each release?).
Cheers,
Thomas Goirand (zigo)
P.S: I'm using puppet-openstack myself, so I wont ever contribute to Kolla and/or OSA, but I still would like to support you...
Hi Thomas,
Thanks for your opinion - but we can’t really wait until each Debian OpenStack release to test all the features (and what will be broken) - we already trail long time after the official OpenStack release due to other issues. That would significantly lengthen the delay and the process involved.
Kolla has been doing voting CI on source images for a long time - and we don’t want to change it.
It’s also a direction to make the external dependencies smaller - not bigger - and I see the custom things added from packager perspective as a dependency.
Best regards, Michal
On 2 Dec 2021, at 01:04, Thomas Goirand zigo@debian.org wrote:
On 11/18/21 1:33 PM, Michał Nasiadka wrote:
Hello Koalas,
On the PTG we have discussed two topics:
- Deprecate and drop binary type of Kolla images
- Use a common base (single Linux distribution) for Kolla images
This is a call for feedback - for people that have not been attending the PTG.
What this essentially mean for consumers:
- In Yoga cycle we will deprecate binary type of Kolla images, and in Z cycle those will be dropped.
- We are not going to support CentOS Stream 9 (cs9) as a base operating system, and the source type build will rely on CentOS Stream 8 in Z release.
- Beginning from A release Kolla will build only Debian source images - but Kolla-Ansible will still support deployment of those images on CentOS/Ubuntu/Debian Host operating systems (and Rocky Linux to be added in Yoga to that mix).
Justification: The Kolla project team is limited in numbers, therefore supporting current broad mix of operating systems (especially with CentOS Stream 9 ,,on the way’’) is a significant maintenance burden. By dropping binary type of images - users would be running more tested images (since Kolla/Kolla-Ansible CI runs source images jobs as voting). In Xena we’ve already changed the default image type Kolla-Ansible uses to source. We also feel that using a unified base OS for Kolla container images is a way to remove some of the maintenance burden (including CI cycles and
Request for feedback: If any of those changes is a no go from your perspective - we’d like to hear your opinions.
Best regards, Michal Nasiadka
Hi,
While I fully support moving to a Debian only world (anyone who knows me isn't surprised to read this...), I really wonder if moving to a "source only" world is the right step.
If you don't have enough people, why don't you just move to a Debian package only type of install, rather than re-implementing all by yourself? The very point of packages is to abstract for you many things, like for example, the fact that Nova has a reserved UID/GID of 64060 in Debian, and will always use that one, and automatically set that up for you, plus put the Nova user in the right group... Or if you need another example, making sure that swift-proxy starts with the right "route = .* addheader:Date: ${httptime[]}" uwsgi parameter... etc.
Now, if you have a limited amount of contributors, all these tiny tweaks that the package maintainers are doing for you, you'll have to re-implement them yourself. Worse: it's going to be even more complicated if you put stuff in containers.
Using source only images, you also don't get all the testing that's done by distro on funny arch which you probably didn't think of. Yes, the openvswitch package in Debian is tested and working on the 10 official Debian arch. The lz4tools contains a 32 bits patch. Or is it that you don't care about anything else than x86_64?
So why don't you just take the other direction, and drop the source images? I'm sure this will save you a lot of work. Plus myself and Michal will make sure everything works, and is available in time for every release, like we always did (did you notice Debian is often (if not always) the first distro to have available packages for each release?).
Cheers,
Thomas Goirand (zigo)
P.S: I'm using puppet-openstack myself, so I wont ever contribute to Kolla and/or OSA, but I still would like to support you...
Hi Michal,
Thanks for your reply.
I'm sorry to say that your reply shows a poor understanding of what distributions and packaging are in general, and what Debian does for OpenStack in particular. But I'm not surprised, this isn't the first time I see this. It also happened when I was employed by Mirantis, and with cloudwatt. People usually just dismiss packaging because they see it as a black-box they cannot act on, because they just happen to not do packaging themselves.
On 12/2/21 8:37 AM, Michał Nasiadka wrote:
Hi Thomas,
Thanks for your opinion - but we can’t really wait until each Debian OpenStack release to test all the features (and what will be broken) - we already trail long time after the official OpenStack release due to other issues. That would significantly lengthen the delay and the process involved.
This isn't fair, and IMO isn't truth.
Usually, Debian packages are ready *before* the final release, often just a week after the first RCs are out. The only reason why there's always one or 2 glitches, is because I couldn't install the new release until everything is finished (and from my side, including updating my puppet scripts for the new release of puppet-openstack).
So, we're talking about just a few days delay after the release. This normally gives you enough time to fix things, and have Kolla ready the day of the release.
I used to package all the beta releases, twice per OpenStack release. Though nobody ever consumed them. And these days, not all projects are release beta releases, so it is kind of useless.
It’s also a direction to make the external dependencies smaller - not bigger - and I see the custom things added from packager perspective as a dependency.
IMO, that's a very wrong perspective. Package maintainers (that's how we're called, not "packager" please...) are adding automations and system integrations. If you refuse them, this means you'll have to implement them yourself. Remember that I use puppet myself, so I very well know what I'm talking about, when discussing config management and package integration. When I can make a choice of where to do the automation, system dependency, etc., then it goes to the packaging: this is the normal and natural place to do it.
A config management system should not be the one taking care of any of the things that packages normally take care of (like dependencies). Handling them in your config management system is not only an heresy, it's also using the wrong tool for it, and achieving a poorer result.
Also, handing dependency management using something like pip is a non-sense, because it only knows about Python dependencies (not anything else), and it's been demonstrated so many times how bad pip is as a dependency solver: for example, it broke the CI so many times, showing its defects.
Another thing you're seeing wrong, is about smaller dependencies. First, having one more Python package will not hurt, and if it is there, it is for a reason. Second, packages are designed to have the smaller amount of dependency possible. Always. As a rule. If there's something wrong there, it's easy to fix (and you could contribute to it...).
And we also make sure that everything works together, adding patches when needed (how are you going to manage patches?).
I'm not hoping to convince you, I feel it's already too late. Though I can't help myself to let you know I believe you're doing the wrong technical choice.
Cheers,
Thomas Goirand (zigo)
On Thu, 2 Dec 2021 at 08:19, Thomas Goirand zigo@debian.org wrote:
Hi Michal,
Thanks for your reply.
I'm sorry to say that your reply shows a poor understanding of what distributions and packaging are in general, and what Debian does for OpenStack in particular. But I'm not surprised, this isn't the first time I see this. It also happened when I was employed by Mirantis, and with cloudwatt. People usually just dismiss packaging because they see it as a black-box they cannot act on, because they just happen to not do packaging themselves.
On 12/2/21 8:37 AM, Michał Nasiadka wrote:
Hi Thomas,
Thanks for your opinion - but we can’t really wait until each Debian OpenStack release to test all the features (and what will be broken) - we already trail long time after the official OpenStack release due to other issues. That would significantly lengthen the delay and the process involved.
This isn't fair, and IMO isn't truth.
Usually, Debian packages are ready *before* the final release, often just a week after the first RCs are out. The only reason why there's always one or 2 glitches, is because I couldn't install the new release until everything is finished (and from my side, including updating my puppet scripts for the new release of puppet-openstack).
So, we're talking about just a few days delay after the release. This normally gives you enough time to fix things, and have Kolla ready the day of the release.
I used to package all the beta releases, twice per OpenStack release. Though nobody ever consumed them. And these days, not all projects are release beta releases, so it is kind of useless.
Currently, we provide 'binary' images for CentOS, Ubuntu and Debian. CentOS provides delorean builds from master, so we can test those on R-26. Ubuntu UCA appears soon after. Only for Debian do we need to wait until ~release. It's not a problem, since we're dropping binary images, I'm just providing some context.
It’s also a direction to make the external dependencies smaller - not bigger - and I see the custom things added from packager perspective as a dependency.
IMO, that's a very wrong perspective. Package maintainers (that's how we're called, not "packager" please...) are adding automations and system integrations. If you refuse them, this means you'll have to implement them yourself. Remember that I use puppet myself, so I very well know what I'm talking about, when discussing config management and package integration. When I can make a choice of where to do the automation, system dependency, etc., then it goes to the packaging: this is the normal and natural place to do it.
A config management system should not be the one taking care of any of the things that packages normally take care of (like dependencies). Handling them in your config management system is not only an heresy, it's also using the wrong tool for it, and achieving a poorer result.
Also, handing dependency management using something like pip is a non-sense, because it only knows about Python dependencies (not anything else), and it's been demonstrated so many times how bad pip is as a dependency solver: for example, it broke the CI so many times, showing its defects.
Another thing you're seeing wrong, is about smaller dependencies. First, having one more Python package will not hurt, and if it is there, it is for a reason. Second, packages are designed to have the smaller amount of dependency possible. Always. As a rule. If there's something wrong there, it's easy to fix (and you could contribute to it...).
And we also make sure that everything works together, adding patches when needed (how are you going to manage patches?).
I'm not hoping to convince you, I feel it's already too late. Though I can't help myself to let you know I believe you're doing the wrong technical choice.
This is all true - we do incur some additional work to build OpenStack from source. OTOH, this has been done and has been stable for many years.
A major benefit we get from source images is being able to point at a local source repo when building images. Thankfully we need to do this less and less frequently as time goes on, but it is still sometimes necessary.
Cheers,
Thomas Goirand (zigo)
participants (9)
-
Adrian Andreias
-
Gregory Orange
-
Marcin Juszkiewicz
-
Mark Goddard
-
Michał Nasiadka
-
Sean Mooney
-
Thomas Goirand
-
Tobias Urdin
-
Tony Liu