[kolla] does someone uses AArch64 images?
AArch64 support in Kolla(-Ansible) project is present since Ocata (iirc). For most of that time we supported all three distributions: CentOS, Debian and Ubuntu. # Distribution coverage CentOS has binary (with packages from RDO project) and source images. We supported version 7, 8 and now we are using CentOS Stream 8 in Wallaby. Debian is source only as there never was any source of up-to-date OpenStack packages (Debian OpenStack team builds only x86-64 packages and they do it after release). Ubuntu has both binary (packages from UCA) and source images. # My interest I take care of Debian/source ones as we use them at Linaro in our developer cloud setups. It often means backporting of fixes when we upgrade from one previous release to something newer. # External repos problem The problem in all cases is external repositories and their lack of support for AArch64 architecture. ## RabbitMQ Biggest problem is with RabbitMQ and Erlang. For x86-64 RMQ developers build packages for CentOS/Debian/Ubuntu. We use them. On AArch64 it differ: - CentOS uses erlang from CentOS distribution - Debian uses erlang from my Linaro OBS repository (where I do builds of backport of Debian 'testing' package). - Ubuntu uses erlang from Ubuntu distribution Ubuntu one is too old - 22.2 while current rabbitmq needs 22.3 or newer. I do not plan to work on getting it working. # Final question I would like to know does anyone use AArch64 images built with Kolla. Please respond which branches you use, distribution used as host OS and as images OS.
On 3/23/21 12:43 PM, Marcin Juszkiewicz wrote:
AArch64 support in Kolla(-Ansible) project is present since Ocata (iirc). For most of that time we supported all three distributions: CentOS, Debian and Ubuntu.
# Distribution coverage
CentOS has binary (with packages from RDO project) and source images. We supported version 7, 8 and now we are using CentOS Stream 8 in Wallaby.
Debian is source only as there never was any source of up-to-date OpenStack packages (Debian OpenStack team builds only x86-64 packages and they do it after release).
Ubuntu has both binary (packages from UCA) and source images.
# My interest
I take care of Debian/source ones as we use them at Linaro in our developer cloud setups. It often means backporting of fixes when we upgrade from one previous release to something newer.
Getting binary support is just a mater of rebuilding packages for arm64. I once did that for you setting-up a Jenkins machine just for rebuilding. It's really a shame that you guys aren't following that road. You'd have all of my support if you did. I personally don't have access to the necessary hardware for it, and wont use the arm64 repos... Cheers, Thomas Goirand (zigo)
W dniu 23.03.2021 o 14:36, Thomas Goirand pisze:
On 3/23/21 12:43 PM, Marcin Juszkiewicz wrote:
AArch64 support in Kolla(-Ansible) project is present since Ocata (iirc). For most of that time we supported all three distributions: CentOS, Debian and Ubuntu.
# Distribution coverage
Debian is source only as there never was any source of up-to-date OpenStack packages (Debian OpenStack team builds only x86-64 packages and they do it after release).
# My interest
I take care of Debian/source ones as we use them at Linaro in our developer cloud setups. It often means backporting of fixes when we upgrade from one previous release to something newer.
Getting binary support is just a mater of rebuilding packages for arm64. I once did that for you setting-up a Jenkins machine just for rebuilding. It's really a shame that you guys aren't following that road. You'd have all of my support if you did. I personally don't have access to the necessary hardware for it, and wont use the arm64 repos...
The thing is - no one asked for Debian/binary images so far so I did not bothered you. Your team provides packages for already released OpenStack stuff. From Kolla point of view it means that at beginning of a new development cycle we would need to get images for previous release with Debian binary packages built, tested and backported.
On 3/29/21 5:34 PM, Marcin Juszkiewicz wrote:
W dniu 23.03.2021 o 14:36, Thomas Goirand pisze:
On 3/23/21 12:43 PM, Marcin Juszkiewicz wrote:
AArch64 support in Kolla(-Ansible) project is present since Ocata (iirc). For most of that time we supported all three distributions: CentOS, Debian and Ubuntu.
# Distribution coverage
Debian is source only as there never was any source of up-to-date OpenStack packages (Debian OpenStack team builds only x86-64 packages and they do it after release).
# My interest
I take care of Debian/source ones as we use them at Linaro in our developer cloud setups. It often means backporting of fixes when we upgrade from one previous release to something newer.
Getting binary support is just a mater of rebuilding packages for arm64. I once did that for you setting-up a Jenkins machine just for rebuilding. It's really a shame that you guys aren't following that road. You'd have all of my support if you did. I personally don't have access to the necessary hardware for it, and wont use the arm64 repos...
The thing is - no one asked for Debian/binary images so far so I did not bothered you.
Your team provides packages for already released OpenStack stuff. From Kolla point of view it means that at beginning of a new development cycle we would need to get images for previous release with Debian binary packages built, tested and backported.
I used to package every beta (b1, b2, b3). But then, nobody was consuming them, so I stopped. And the harsh reality right now, is that most projects stopped producing these bX releases. Today, I'm already done with the packaging of all RC1 releases, and I'm already beginning to test installing them. This is 2 days after the RCs. No other distribution beats me on this timing. So I'm not sure what more I could do... If you wish to participate and provide intermediary releases using Git, that's possible, but that means someone needs to participate and do it. I can't be full time doing packaging like I used to when I was employed to do only that (these days, I maintain my own installer and 8 clusters in production...). Radosław Piliszek often ask me about the status of Debian packages, and he manages to maintain compat in OSA using them. Why can't you do it? Again, I very much would love to collaborate and help you doing more with the Debian binary stuff for ARM. Cheers, Thomas Goirand (zigo)
On Mon, Mar 29, 2021 at 7:49 PM Thomas Goirand <zigo@debian.org> wrote:
Radosław Piliszek often ask me about the status of Debian packages, and he manages to maintain compat in OSA using them. Why can't you do it?
Just for the record, I do it for Kolla too. :-) Just x86_64, not aarch64. -yoctozepto
W dniu 29.03.2021 o 19:48, Thomas Goirand pisze:
On 3/29/21 5:34 PM, Marcin Juszkiewicz wrote:
Getting binary support is just a mater of rebuilding packages for arm64. I once did that for you setting-up a Jenkins machine just for rebuilding. It's really a shame that you guys aren't following that road. You'd have all of my support if you did. I personally don't have access to the necessary hardware for it, and wont use the arm64 repos...
The thing is - no one asked for Debian/binary images so far so I did not bothered you.
Your team provides packages for already released OpenStack stuff. From Kolla point of view it means that at beginning of a new development cycle we would need to get images for previous release with Debian binary packages built, tested and backported.
I used to package every beta (b1, b2, b3). But then, nobody was consuming them, so I stopped. And the harsh reality right now, is that most projects stopped producing these bX releases.
Today, I'm already done with the packaging of all RC1 releases, and I'm already beginning to test installing them. This is 2 days after the RCs.
Are they available somewhere online so they can be tested?
No other distribution beats me on this timing.
RDO and UCA produce packages during development cycle.
So I'm not sure what more I could do... If you wish to participate and provide intermediary releases using Git, that's possible, but that means someone needs to participate and do it. I can't be full time doing packaging like I used to when I was employed to do only that (these days, I maintain my own installer and 8 clusters in production...).
I am aware that building OpenStack packages is not your daily job anymore.
Radosław Piliszek often ask me about the status of Debian packages, and he manages to maintain compat in OSA using them. Why can't you do it?
Taking care of Kolla is one of my work tasks. Not primary anymore as we got to the point when I only need to fix when things break.
Again, I very much would love to collaborate and help you doing more with the Debian binary stuff for ARM. I can check are there free resources left for providing you AArch64 machine for builds.
On 3/30/21 3:20 PM, Marcin Juszkiewicz wrote:
W dniu 29.03.2021 o 19:48, Thomas Goirand pisze:
On 3/29/21 5:34 PM, Marcin Juszkiewicz wrote:
Getting binary support is just a mater of rebuilding packages for arm64. I once did that for you setting-up a Jenkins machine just for rebuilding. It's really a shame that you guys aren't following that road. You'd have all of my support if you did. I personally don't have access to the necessary hardware for it, and wont use the arm64 repos...
The thing is - no one asked for Debian/binary images so far so I did not bothered you.
Your team provides packages for already released OpenStack stuff. From Kolla point of view it means that at beginning of a new development cycle we would need to get images for previous release with Debian binary packages built, tested and backported.
I used to package every beta (b1, b2, b3). But then, nobody was consuming them, so I stopped. And the harsh reality right now, is that most projects stopped producing these bX releases.
Today, I'm already done with the packaging of all RC1 releases, and I'm already beginning to test installing them. This is 2 days after the RCs.
Are they available somewhere online so they can be tested?
Of course: either through extrepo (for example: "extrepo enable openstack_wallaby") or directly at: http://osbpo.debian.net/debian/ Look into the "dists" folder to see all what's available: at the moment, all from jessie-liberty to bullseye-wallaby.
So I'm not sure what more I could do... If you wish to participate and provide intermediary releases using Git, that's possible, but that means someone needs to participate and do it. I can't be full time doing packaging like I used to when I was employed to do only that (these days, I maintain my own installer and 8 clusters in production...).
I am aware that building OpenStack packages is not your daily job anymore.
It is part of my daily job, but not 100% of my time. Maybe 20%...
Again, I very much would love to collaborate and help you doing more with the Debian binary stuff for ARM. I can check are there free resources left for providing you AArch64 machine for builds.
I'm ok to do the setup, and explain how to use the box, however, someone else will have to babysit the build process: I don't want to just do it for free, with no return, when I'm not using ARM boxes myself. Note that I need one Jenkins builder VM per OpenStack release, and that the setup is done using Ansible. Cheers, Thomas Goirand (zigo)
W dniu 30.03.2021 o 21:30, Thomas Goirand pisze:
On 3/30/21 3:20 PM, Marcin Juszkiewicz wrote:
Today, I'm already done with the packaging of all RC1 releases, and I'm already beginning to test installing them. This is 2 days after the RCs.
Are they available somewhere online so they can be tested?
Of course: either through extrepo (for example: "extrepo enable openstack_wallaby") or directly at: http://osbpo.debian.net/debian/
Look into the "dists" folder to see all what's available: at the moment, all from jessie-liberty to bullseye-wallaby.
While we are buster-wallaby ;D Move to bullseye will happen once Xena cycle opens - I waited for full freeze with it and had several other things to do.
Again, I very much would love to collaborate and help you doing more with the Debian binary stuff for ARM. I can check are there free resources left for providing you AArch64 machine for builds.
I'm ok to do the setup, and explain how to use the box, however, someone else will have to babysit the build process: I don't want to just do it for free, with no return, when I'm not using ARM boxes myself.
Understood.
Note that I need one Jenkins builder VM per OpenStack release, and that the setup is done using Ansible.
Will ask around.
On 3/31/21 9:34 AM, Marcin Juszkiewicz wrote:
W dniu 30.03.2021 o 21:30, Thomas Goirand pisze:
On 3/30/21 3:20 PM, Marcin Juszkiewicz wrote:
Today, I'm already done with the packaging of all RC1 releases, and I'm already beginning to test installing them. This is 2 days after the RCs.
Are they available somewhere online so they can be tested?
Of course: either through extrepo (for example: "extrepo enable openstack_wallaby") or directly at: http://osbpo.debian.net/debian/
Look into the "dists" folder to see all what's available: at the moment, all from jessie-liberty to bullseye-wallaby.
While we are buster-wallaby ;D Move to bullseye will happen once Xena cycle opens - I waited for full freeze with it and had several other things to do.
Well, that's not how we do things in the Debian OpenStack team. We generally have one single OpenStack release which we both build for stable and stable+1. This release is Victoria. So we do have non-official repositories for Victoria under Buster and Bullseye: - buster-victoria - bullseye-victoria The reason we do that, is that we want our users to be able to switch Debian release without changing OpenStack release. Wallaby is currently pushed to Debian Experimental, and we will only be able to upload to unstable once Bullseye is released, which normally should happen before Xena. Hopefully, you guys at Linaro can align with this policy?
Again, I very much would love to collaborate and help you doing more with the Debian binary stuff for ARM. I can check are there free resources left for providing you AArch64 machine for builds.
I'm ok to do the setup, and explain how to use the box, however, someone else will have to babysit the build process: I don't want to just do it for free, with no return, when I'm not using ARM boxes myself.
Understood.
Just one more thing: of course, I can do the first bootstraping to make sure we have at least Victoria + Wallaby. If you didn't know, we have a status page here: http://osbpo.debian.net/deb-status/ This isn't really official, this is more some private tooling so we know what work we need to do. (This was written by Michal Arbet and his brother Jan, both from ultimum.io.) Hopefully, we can also add ARM support there. Also, have you tried Victoria with plain Bullseye? It should work out of the box from the Debian official repositories normally... That's the good thing about this timing: there shouldn't be much to backport to an ARM specific repository since we're just after the freeze of Bullseye.
Note that I need one Jenkins builder VM per OpenStack release, and that the setup is done using Ansible.
Will ask around.
Once you have the VMs, I can make debian.net point to it. I generally setup a CNAME pointer. This is important so we can setup a letsencrypt SSL certificate for the Jenkins URLs. BTW, what kind of 1U / 2U servers would you recommend for ARM in production, for storage and/or compute workload? Cheers, Thomas Goirand (zigo)
W dniu 31.03.2021 o 10:15, Thomas Goirand pisze:
On 3/31/21 9:34 AM, Marcin Juszkiewicz wrote:
Of course: either through extrepo (for example: "extrepo enable openstack_wallaby") or directly at: http://osbpo.debian.net/debian/
Look into the "dists" folder to see all what's available: at the moment, all from jessie-liberty to bullseye-wallaby.
While we are buster-wallaby ;D Move to bullseye will happen once Xena cycle opens - I waited for full freeze with it and had several other things to do.
Well, that's not how we do things in the Debian OpenStack team. We generally have one single OpenStack release which we both build for stable and stable+1. This release is Victoria. So we do have non-official repositories for Victoria under Buster and Bullseye:
- buster-victoria - bullseye-victoria
The reason we do that, is that we want our users to be able to switch Debian release without changing OpenStack release.
Wallaby is currently pushed to Debian Experimental, and we will only be able to upload to unstable once Bullseye is released, which normally should happen before Xena.
Hopefully, you guys at Linaro can align with this policy?
We only use Kolla images to deploy. And Wallaby Kolla is on Buster. Linaro cloud is on Ussuri still iirc. Doing setup using distro packages is something we had during Liberty times. Then moved to some weird in-house solution and then to Kolla containers (using source).
Also, have you tried Victoria with plain Bullseye? It should work out of the box from the Debian official repositories normally...
All our setup are done with Kolla containers.
BTW, what kind of 1U / 2U servers would you recommend for ARM in production, for storage and/or compute workload?
Kevin Zhao (cc:) would probably be better on hardware recommendations. I mostly complain that we still use v8.0 (2012 tech) ones ;D
participants (3)
-
Marcin Juszkiewicz
-
Radosław Piliszek
-
Thomas Goirand