Thanks for responding Alex. On Mon, 25 Nov 2019 at 18:04, Alex Schultz <aschultz@redhat.com> wrote:

On Mon, Nov 25, 2019 at 10:54 AM Mark Goddard <mark@stackhpc.com> wrote:

...

Hi,

During the recent kolla PTG discussions [1], we covered CentOS 8 in some detail. For kolla there are several aspects to CentOS 8 support:

* kolla-build execution host * container images * kolla-ansible execution host * kolla-ansible remote hosts

In this context I am discussing the last of these - kolla-ansible remote hosts, i.e. the OS on the controllers and computes etc. Given that this could be quite a thorny problem, I'd like to propose that we collaborate between deployment projects.

It is my understanding that upgrades from CentOS 7 to 8 will not be supported, and a reinstall is required. We set out some goals for the migration:

* Migrate hosts from CentOS 7 to CentOS 8 * Upgrade from Train CentOS 7 containers to Ussuri CentOS 8 containers * Note: this means Ussuri containers (CentOS 8) don't land on CentOS 7 * Decouple the CentOS 7 to 8 and OpenStack upgrades, to avoid operator pain * Avoid excessive downtime

There is a Red Hat policy [2] that suggests that if you mix host and container OS versions, it is safer for the host to be ahead of the container. This makes sense if you think about userland accessing kernel features.

This leads us to the following migration path:

* Start with Train release, CentOS 7 containers * Redeploy (rolling) hosts with CentOS 8 * Provision * Configure host OS * Redeploy Train containers with CentOS 7

* Upgrade to Ussuri release, CentOS 8 containers

The logic we used to come up with this procedure is as follows:

* Host must be upgraded to CentOS 7 before containers (going on policy) * Train containers don't support CentOS 8 base due to RDO

I believe the plan is to have a Train version on CentOS8 after all the things get bootstrapped. Unfortunately the current target is trying to get master on centos8 with the time frame currently TBD. I'm personally hoping really soon.

Hopefully it won't require too many changes in kolla to support it. Based on this we could modify our plan to use Train containers with a CentOS 8 base when deploying on CentOS 8 hosts. We'd need to support building and publishing both CentOS 7 and 8 containers for the Train cycle. That would avoid the host/container mismatch.

...

* Ussuri containers don't support CentOS 7 base due to RDO * Want to separate CentOS 7 -> 8 migration from OpenStack upgrade

This leads us to the conclusion that at least one release of kolla-ansible must bridge these worlds, since we will be deploying Train containers on CentOS 8. We could do either of:

1. Train k-a supports CentOS 8 hosts 2. Ussuri k-a supports deploying Train containers

While 1. may require us to backport more than we would like, supporting deployment of multiple OpenStack releases could be challenging, so we expect to go with 1 here.

There are some risks involved, including:

* some Train services may not work on CentOS 8 * ceph? * OVS & neutron agents? * libvirt? * Ansible minimum version 2.8 for CentOS 8 (ref?)

These should be fine as we're testing these with actual rhel8 over in TripleO (at least builds/running) for Train+. You'll run into other issues with pacemaker if you currently use that and try to mix versions.

I think with a CentOS 8 based container I'd be less worried here. Also if Tripleo needs this to work :)

Bigger risk: lack of docker packaging in the upstream which also means no docker-distribution for local repositories.

Currently for testing CentOS 8 we are using the CentOS 7 Docker upstream repo, which requires module_hotfixes to work. https://review.opendev.org/#/c/692794/4/tools/setup_RedHat.sh

...

Hopefully we can save ourselves some effort and solve this together.

[1] https://etherpad.openstack.org/p/kolla-ussuri-ptg [2] https://access.redhat.com/support/policy/rhel-container-compatibility

Thanks, Mark

On Tue, 26 Nov 2019 at 11:30, Jonathan Rosser <jonathan.rosser@rd.bbc.co.uk> wrote:

On 25/11/2019 17:48, Mark Goddard wrote:

...

Hi,

During the recent kolla PTG discussions [1], we covered CentOS 8 in some detail. For kolla there are several aspects to CentOS 8 support:

* kolla-build execution host * container images * kolla-ansible execution host * kolla-ansible remote hosts

Hi Mark,

Whilst the container approach in openstack-ansible is different (LXC) and presents it's own challenges for Centos8, there is no doubt a common set of issues. We also do a container-less bare-metal deploy mode which would be the first to try to get working.

I have a WIP patch for OSA for Centos-8 for a while now which really does not get so far.

https://review.opendev.org/#/c/689629

I've not yet managed to create dummy interfaces with network-manager which leads to a horrible hack setting up the networking for CI.

Also failing was building of dbus-python which is required for the Ansible nmcli module, due to an incompatibility with autotools 1.16.

There are a also few pieces we take from EPEL (lsyncd) which are missing.

Then there is/was the absence of an obvious source of RDO packages.

I am concerned about the difficulty of creating a tractable upgrade path which separates OS from OpenStack upgrades. For openstack-ansible, Rocky was the transition release which supported both Xenial and Bionic. Achieving the same transition release for Centos 7 to 8 upgrades looks challenging.

Jon.

Thanks Jon, there's some useful information in there. Given that RDO Ussuri will not support CentOS 7, I think you will also need to do the 7 to 8 transition on Train. The timing's not ideal here - I can see all the deployment tools either needing to backport large changes to stable/train, or delay their releases (if they haven't released yet). Let's keep this thread going as we each hit different roadblocks.