Open Stack

There hasn't been a Python 2 release in 8 years, and during that time 
we've gotten used to the idea that that's the way things go. However, 
with the switch to Python 3 looming (we will drop support for Python 2 
in the U release[1]), history is no longer a good guide: Python 3 
releases drop as often as every year. We are already feeling the pain 
from this, as Linux distros have largely already completed the shift to 
Python 3, and those that have are on versions newer than the py35 we 
currently have in gate jobs.

We have traditionally held to the principle that we want each release to 
support the latest release of CentOS and the latest LTS release of 
Ubuntu, as they existed at the beginning of the release cycle.[2] 
Currently this means in practice one version of py2 and one of py3, but 
in the future it will mean two, usually different, versions of py3.

There are two separate issues that we need to address: unit tests (we'll 
define this as code tested in isolation, within or spawned from within 
the testing process), and integration tests (we'll define this as code 
running in its own process, tested from the outside). I have two 
separate but related proposal for how to handle those.

I'd like to avoid discussion which versions of things we think should be 
supported in Stein in this thread. Let's come up with a process that we 
think is a good one to take into T and beyond, and then retroactively 
apply it to Stein. Competing proposals are of course welcome, in 
addition to feedback on this one.

Unit Tests
----------

For unit tests, the most important thing is to test on the versions of 
Python we target. It's less important to be using the exact distro that 
we want to target, because unit tests generally won't interact with 
stuff outside of Python.

I'd like to propose that we handle this by setting up a unit test 
template in openstack-zuul-jobs for each release. So for Stein we'd have 
openstack-python3-stein-jobs. This template would contain:

* A voting gate job for the highest minor version of py3 we want to 
support in that release.
* A voting gate job for the lowest minor version of py3 we want to 
support in that release.
* A periodic job for any interim minor releases.
* (Starting late in the cycle) a non-voting check job for the highest 
minor version of py3 we want to support in the *next* release (if 
different), on the master branch only.

So, for example, (and this is still under active debate) for Stein we 
might have gating jobs for py35 and py37, with a periodic job for py36. 
The T jobs might only have voting py36 and py37 jobs, but late in the T 
cycle we might add a non-voting py38 job on master so that people who 
haven't switched to the U template yet can see what, if anything, 
they'll need to fix.

We'll run the unit tests on any distro we can find that supports the 
version of Python we want. It could be a non-LTS Ubuntu, Fedora, Debian 
unstable, whatever it takes. We won't wait for an LTS Ubuntu to have a 
particular Python version before trying to test it.

Before the start of each cycle, the TC would determine which range of 
versions we want to support, on the basis of the latest one we can find 
in any distro and the earliest one we're likely to need in one of the 
supported Linux distros. There will be a project-wide goal to switch the 
testing template from e.g. openstack-python3-stein-jobs to 
openstack-python3-treasure-jobs for every repo before the end of the 
cycle. We'll have goal champions as usual following up and helping teams 
with the process. We'll know where the problem areas are because we'll 
have added non-voting jobs for any new Python versions to the previous 
release's template.

Integration Tests
-----------------

Integration tests do test, amongst other things, integration with 
non-openstack-supplied things in the distro, so it's important that we 
test on the actual distros we have identified as popular.[2] It's also 
important that every project be testing on the same distro at the end of 
a release, so we can be sure they all work together for users.

When a new release of CentOS or a new LTS release of Ubuntu comes out, 
the TC will create a project-wide goal for the *next* release cycle to 
switch all integration tests over to that distro. It's up to individual 
projects to make the switch for the tests that they own (e.g. it'd be 
the QA team for Tempest, but other individual projects for their own 
jobs). Again, there'll be a goal champion to monitor and follow up.


[1] 
https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation-timeline.html
[2] 
https://governance.openstack.org/tc/reference/project-testing-interface.html#linux-distributions