Thanks for responding Alex.
On Mon, 25 Nov 2019 at 18:04, Alex Schultz aschultz@redhat.com wrote:
On Mon, Nov 25, 2019 at 10:54 AM Mark Goddard mark@stackhpc.com wrote:
Hi,
During the recent kolla PTG discussions [1], we covered CentOS 8 in some detail. For kolla there are several aspects to CentOS 8 support:
- kolla-build execution host
- container images
- kolla-ansible execution host
- kolla-ansible remote hosts
In this context I am discussing the last of these - kolla-ansible remote hosts, i.e. the OS on the controllers and computes etc. Given that this could be quite a thorny problem, I'd like to propose that we collaborate between deployment projects.
It is my understanding that upgrades from CentOS 7 to 8 will not be supported, and a reinstall is required. We set out some goals for the migration:
- Migrate hosts from CentOS 7 to CentOS 8
- Upgrade from Train CentOS 7 containers to Ussuri CentOS 8 containers
- Note: this means Ussuri containers (CentOS 8) don't land on CentOS 7
- Decouple the CentOS 7 to 8 and OpenStack upgrades, to avoid operator pain
- Avoid excessive downtime
There is a Red Hat policy [2] that suggests that if you mix host and container OS versions, it is safer for the host to be ahead of the container. This makes sense if you think about userland accessing kernel features.
This leads us to the following migration path:
Start with Train release, CentOS 7 containers
Redeploy (rolling) hosts with CentOS 8
- Provision
- Configure host OS
- Redeploy Train containers with CentOS 7
Upgrade to Ussuri release, CentOS 8 containers
The logic we used to come up with this procedure is as follows:
- Host must be upgraded to CentOS 7 before containers (going on policy)
- Train containers don't support CentOS 8 base due to RDO
I believe the plan is to have a Train version on CentOS8 after all the things get bootstrapped. Unfortunately the current target is trying to get master on centos8 with the time frame currently TBD. I'm personally hoping really soon.
Hopefully it won't require too many changes in kolla to support it. Based on this we could modify our plan to use Train containers with a CentOS 8 base when deploying on CentOS 8 hosts. We'd need to support building and publishing both CentOS 7 and 8 containers for the Train cycle. That would avoid the host/container mismatch.
- Ussuri containers don't support CentOS 7 base due to RDO
- Want to separate CentOS 7 -> 8 migration from OpenStack upgrade
This leads us to the conclusion that at least one release of kolla-ansible must bridge these worlds, since we will be deploying Train containers on CentOS 8. We could do either of:
- Train k-a supports CentOS 8 hosts
- Ussuri k-a supports deploying Train containers
While 1. may require us to backport more than we would like, supporting deployment of multiple OpenStack releases could be challenging, so we expect to go with 1 here.
There are some risks involved, including:
- some Train services may not work on CentOS 8
- ceph?
- OVS & neutron agents?
- libvirt?
- Ansible minimum version 2.8 for CentOS 8 (ref?)
These should be fine as we're testing these with actual rhel8 over in TripleO (at least builds/running) for Train+. You'll run into other issues with pacemaker if you currently use that and try to mix versions.
I think with a CentOS 8 based container I'd be less worried here. Also if Tripleo needs this to work :)
Bigger risk: lack of docker packaging in the upstream which also means no docker-distribution for local repositories.
Currently for testing CentOS 8 we are using the CentOS 7 Docker upstream repo, which requires module_hotfixes to work. https://review.opendev.org/#/c/692794/4/tools/setup_RedHat.sh
Hopefully we can save ourselves some effort and solve this together.
[1] https://etherpad.openstack.org/p/kolla-ussuri-ptg [2] https://access.redhat.com/support/policy/rhel-container-compatibility
Thanks, Mark