On Mon, Nov 25, 2019 at 10:54 AM Mark Goddard <mark@stackhpc.com> wrote:
Hi,

During the recent kolla PTG discussions [1], we covered CentOS 8 in
some detail. For kolla there are several aspects to CentOS 8 support:

* kolla-build execution host
* container images
* kolla-ansible execution host
* kolla-ansible remote hosts

In this context I am discussing the last of these - kolla-ansible
remote hosts, i.e. the OS on the controllers and computes etc. Given
that this could be quite a thorny problem, I'd like to propose that we
collaborate between deployment projects.

It is my understanding that upgrades from CentOS 7 to 8 will not be
supported, and a reinstall is required. We set out some goals for the
migration:

* Migrate hosts from CentOS 7 to CentOS 8
* Upgrade from Train CentOS 7 containers to Ussuri CentOS 8 containers
  * Note: this means Ussuri containers (CentOS 8) don't land on CentOS 7
* Decouple the CentOS 7 to 8 and OpenStack upgrades, to avoid operator pain
* Avoid excessive downtime

There is a Red Hat policy [2] that suggests that if you mix host and
container OS versions, it is safer for the host to be ahead of the
container. This makes sense if you think about userland accessing
kernel features.

This leads us to the following migration path:

* Start with Train release, CentOS 7 containers
* Redeploy (rolling) hosts with CentOS 8
  * Provision
  * Configure host OS
  * Redeploy Train containers with CentOS 7
* Upgrade to Ussuri release, CentOS 8 containers

The logic we used to come up with this procedure is as follows:

* Host must be upgraded to CentOS 7 before containers (going on policy)
* Train containers don't support CentOS 8 base due to RDO

I believe the plan is to have a Train version on CentOS8 after all the things get bootstrapped. Unfortunately the current target is trying to get master on centos8 with the time frame currently TBD.  I'm personally hoping really soon.
 
* Ussuri containers don't support CentOS 7 base due to RDO
* Want to separate CentOS 7 -> 8 migration from OpenStack upgrade

This leads us to the conclusion that at least one release of
kolla-ansible must bridge these worlds, since we will be deploying
Train containers on CentOS 8. We could do either of:

1. Train k-a supports CentOS 8 hosts
2. Ussuri k-a supports deploying Train containers

While 1. may require us to backport more than we would like,
supporting deployment of multiple OpenStack releases could be
challenging, so we expect to go with 1 here.

There are some risks involved, including:

* some Train services may not work on CentOS 8
  * ceph?
  * OVS & neutron agents?
  * libvirt?
* Ansible minimum version 2.8 for CentOS 8 (ref?)

These should be fine as we're testing these with actual rhel8 over in TripleO (at least builds/running) for Train+.  You'll run into other issues with pacemaker if you currently use that and try to mix versions.

Bigger risk: lack of docker packaging in the upstream which also means no docker-distribution for local repositories.

 

Hopefully we can save ourselves some effort and solve this together.

[1] https://etherpad.openstack.org/p/kolla-ussuri-ptg
[2] https://access.redhat.com/support/policy/rhel-container-compatibility

Thanks,
Mark