Open Stack

Hi,

This week we held a virtual PTG. The etherpad [1] includes the agenda
and notes. Here is my summary of what was discussed, decisions and
actions.

# Cross-project

We started the sessions with a visitor from Tripleo - Tengu - who is
working on validations [2].

Validations are a way of checking a control plane at various points in
its life cycle, usually to check that some operation was completed
successfully. Tengu described the project and explained how there was
interest in making it a more generic framework that could be used by
kolla ansible and other deployment projects. On our side there was no
objection but it would require an owner. I offered to help Tengu with
a PoC to see how it would work.

# General

Next we moved onto general project matters.

##  Sustainability

We've seen a slow decline in contribution over the past few cycles, as
have many other projects. Part of this is natural for a more mature
project that now mostly does what is required of it. Still, its good
to reflect on sustainability, and plan ahead for leaner times. As a
project we have a huge support matrix, with hundreds of container
images, multiple OS distributions, different CPU architectures and a
plethora of configuration options. We decided that a good starting
point would be to define some categories of support:

* images we maintain
* images we more or less care about
* the rest (once broken we may or may not work on fixing)

The wording may need some work :) We can extend this also to OS
distros, CPU arches, service deployment, and features. A key factor
here will be the level of test coverage. We would document this on the
wiki or docs.o.o. Ultimately this is all best effort, given that we're
an open source project with a community of volunteers.

A way to improve sustainability we discussed is through trimming the
feature matrix.

* OracleLinux - this is small but non-zero maintenance overhead. It's
a candidate because we don't see people using it and Oracle
maintainers left the project.
* Debian binary images - not seeing many users of these.
* Ceph - this is well used but does require maintenance. We support
integration with external ceph clusters, and have discussed switching
to recommending ceph-ansible, with a documented, automated and tested
migration path.
* kolla-cli - this project was added a few cycles ago, then the
maintainers left the community. We haven't released it, CI is broken
and have heard no complaints.

For features which do not require much code we could disable testing,
and move them to an explicit 'unmaintained' status. Alternatively, we
could deprecate then remove them. I will follow up with emails asking
for community feedback on the impact of removal.

## Core team

We're looking out for potential core reviewers. If you'd like to join
the team, speak to one of us and we can help you on the path. The main
factor here is quality, thoughtful reviews.

## Meetings

Recently attendance at IRC meetings has been low. We decided to move
the meetings to #openstack-kolla to try to get more people involved.

## Kayobe

Kayobe seeks to become an official project. This could be under kolla
project governance, or as a separate project - the main deciding
factor will be the kolla team. I will send an email about this
separately.

## Priorities

I would like us to agree as a team on some priorities for the Train
cycle. I will send out a separate mail about this.

# Kolla (images)

## Python 3 images

In the Train cycle all projects should be moving to support python 3
only. We therefore need to build images based on python 3. hrw has
been working on this [3] for Ubuntu and Debian source images.

We will also need to switch to python 3 for CentOS/RHEL source images.
This work needs an owner. A related issue is that of CentOS 8, which
will support only python 3. I will follow up with a mail about this.

Binary images (RPM/deb) depend on the distros and their plans for python 3.

There is some python 3 work in kolla-ansible, to ensure we can execute
ansible via python both locally and on remote hosts.

## Health checks

Tripleo builds some healthchecks [4] into their docker images. This
allows Docker to restart a service if it is deemed to be unhealthy. We
agreed it would be nice to see support for this in kolla but did not
discuss in depth.

## Ansible version upgrade

Ansible moves on, and so must we. The version of ansible in
kolla_toolbox is now rather old (2.2), and our minimum ansible version
for kolla-ansible (2.5) is also getting old. It is likely we will need
to update some of the custom ansible modules in order to do this. We
may be able to replace others with modules from Ansible (e.g.
kolla_keystone_user, kolla_keystone_service). mnasiadka offered to
pick this up after the Ceph Nautilus upgrade.

## Fluentd upgrade

The fluentd service needs an update - we are  using 0.12.something.
Needs an owner.

## Machine readable image config

The issues this would solve have now mostly been fixed in
tools/version-check.py, so we'll probably leave it.

## Buildah support

At the Denver summit there was some interest in support for buildah as
an alternative to docker for building images. I am led to believe
tripleo already does this, so will ask how they do it.

# Kolla Ansible

## Test coverage

We made some good progress on improving CI test coverage during the
Stein cycle, adding these jobs:

* Cinder LVM
* Scenario NFV
* Major version upgrades
* Zun

During the Train cycle we aim to add these:

* MariaDB: https://review.opendev.org/655663
* Monasca: https://review.opendev.org/649893 (WIP)
* Ironic: https://review.opendev.org/568829
* Ceph upgrade: https://review.opendev.org/658132
* Tempest: https://review.opendev.org/402122 (WIP)

Other candidates include magnum, octavia and prometheus.

## Nova cells v2

We had a long discussion about nova cells v2 [5]. Thanks to jroll and
johnthetubaguy for getting involved. We generally agreed that this
should be part of a wider assessment of scalability in kolla-ansible,
although I was keen to treat different aspects of this separately
during development.

There are a number of ways to approach a multi-cell deployment,
particularly in relation to how the per-cell infrastructure (DB, MQ,
nova-conductor) are deployed. We discussed building a flexible
mechanism for stamping out this infrastructure to arbitrary locations,
then being able to point services in each cell at a given
infrastructure location.

In an effort to make this as simple to use for deployers as possible,
we discussed defining a reference large scale cloud architecture. Our
first pass at this is as follows:

* API controllers x3+: APIs, super conductors, Galera, RabbitMQ, Keystone, etc
* Cell controller clusters x3+: cell conductors, Galera, RabbitMQ,
Glance, maybe neutron? (or one per "failure domain / AZ")
* Cell computes: nova compute, neutron agents

We would aim to add a CI job based on a two cell cloud using this architecture.

We moved on to operations in a multi-cell world. It should be easy to
add API controllers, cell controllers and compute nodes. We would want
the ability to operate on the cloud as a whole, or individual cells
using --limit.

Upgrades seem likely to introduce challenges, particularly if we want
to upgrade cells individually. It's likely CERN and others have some
good experience we could benefit from here.

The spec [6] needs an update based on this discussion, but comments
there are welcome.

## Others

We ran out of time before discussing the other kolla-ansible issues.
Please update the etherpad [1] if you have thoughts on any of these.

# Thanks

Thanks to everyone who attended the kolla PTG. A conference call is
certainly more challenging than a face to face dIscussion, but it was
also nice to not have to fly people around the world for a few design
discussions. I feel we did make progress on a number of important
issues. If anyone has feedback on how we could improve next time,
please get in touch.

Cheers,
Mark

[1] https://etherpad.openstack.org/p/kolla-train-ptg
[2] https://docs.openstack.org/tripleo-validations/latest/readme.html
[3] https://review.opendev.org/642375
[4] https://github.com/openstack/tripleo-common/tree/master/healthcheck
[5] https://blueprints.launchpad.net/kolla-ansible/+spec/support-nova-cells
[6] https://review.openstack.org/616645