Open Stack

On Fri, Oct 19, 2018, at 8:17 AM, Zane Bitter wrote:
> There hasn't been a Python 2 release in 8 years, and during that time 
> we've gotten used to the idea that that's the way things go. However, 
> with the switch to Python 3 looming (we will drop support for Python 2 
> in the U release[1]), history is no longer a good guide: Python 3 
> releases drop as often as every year. We are already feeling the pain 
> from this, as Linux distros have largely already completed the shift to 
> Python 3, and those that have are on versions newer than the py35 we 
> currently have in gate jobs.
> 
> We have traditionally held to the principle that we want each release to 
> support the latest release of CentOS and the latest LTS release of 
> Ubuntu, as they existed at the beginning of the release cycle.[2] 
> Currently this means in practice one version of py2 and one of py3, but 
> in the future it will mean two, usually different, versions of py3.
> 
> There are two separate issues that we need to address: unit tests (we'll 
> define this as code tested in isolation, within or spawned from within 
> the testing process), and integration tests (we'll define this as code 
> running in its own process, tested from the outside). I have two 
> separate but related proposal for how to handle those.
> 
> I'd like to avoid discussion which versions of things we think should be 
> supported in Stein in this thread. Let's come up with a process that we 
> think is a good one to take into T and beyond, and then retroactively 
> apply it to Stein. Competing proposals are of course welcome, in 
> addition to feedback on this one.
> 
> Unit Tests
> ----------
> 
> For unit tests, the most important thing is to test on the versions of 
> Python we target. It's less important to be using the exact distro that 
> we want to target, because unit tests generally won't interact with 
> stuff outside of Python.
> 
> I'd like to propose that we handle this by setting up a unit test 
> template in openstack-zuul-jobs for each release. So for Stein we'd have 
> openstack-python3-stein-jobs. This template would contain:

Because zuul config is branch specific we could set up every project to use a `openstack-python3-jobs` template then define that template differently on each branch. This would mean you only have to update the location where the template is defined and not need to update every other project after cutting a stable branch. I would suggest we take advantage of that to reduce churn.

> 
> * A voting gate job for the highest minor version of py3 we want to 
> support in that release.
> * A voting gate job for the lowest minor version of py3 we want to 
> support in that release.
> * A periodic job for any interim minor releases.
> * (Starting late in the cycle) a non-voting check job for the highest 
> minor version of py3 we want to support in the *next* release (if 
> different), on the master branch only.
> 
> So, for example, (and this is still under active debate) for Stein we 
> might have gating jobs for py35 and py37, with a periodic job for py36. 
> The T jobs might only have voting py36 and py37 jobs, but late in the T 
> cycle we might add a non-voting py38 job on master so that people who 
> haven't switched to the U template yet can see what, if anything, 
> they'll need to fix.
> 
> We'll run the unit tests on any distro we can find that supports the 
> version of Python we want. It could be a non-LTS Ubuntu, Fedora, Debian 
> unstable, whatever it takes. We won't wait for an LTS Ubuntu to have a 
> particular Python version before trying to test it.
> 
> Before the start of each cycle, the TC would determine which range of 
> versions we want to support, on the basis of the latest one we can find 
> in any distro and the earliest one we're likely to need in one of the 
> supported Linux distros. There will be a project-wide goal to switch the 
> testing template from e.g. openstack-python3-stein-jobs to 
> openstack-python3-treasure-jobs for every repo before the end of the 
> cycle. We'll have goal champions as usual following up and helping teams 
> with the process. We'll know where the problem areas are because we'll 
> have added non-voting jobs for any new Python versions to the previous 
> release's template.

I don't know that this needs to be a project wide goal if you can just update the template on the master branch where the template is defined. Do that then every project is now running with the up to date version of the template. We should probably advertise when this is happening with some links to python version x.y breakages/features, but the process itself should be quick.

As for python version range selection I worry that that the criteria about relies on too much guesswork. I do think we should do our best to test future incoming versions of python even while not officially supporting them. We will have to support them at some point, either directly or via some later version that includes the changes from that intermediate version. Could the criteria be:
Support the lowest version of python on a supported distro release and the highest version of python on a supported distro and test (but not support so we can drop testing for this python version on stable branches) the current latest release of python?

This is objective and doesn't require anyone to guess at what versions need to be supported.

> 
> Integration Tests
> -----------------
> 
> Integration tests do test, amongst other things, integration with 
> non-openstack-supplied things in the distro, so it's important that we 
> test on the actual distros we have identified as popular.[2] It's also 
> important that every project be testing on the same distro at the end of 
> a release, so we can be sure they all work together for users.
> 
> When a new release of CentOS or a new LTS release of Ubuntu comes out, 
> the TC will create a project-wide goal for the *next* release cycle to 
> switch all integration tests over to that distro. It's up to individual 
> projects to make the switch for the tests that they own (e.g. it'd be 
> the QA team for Tempest, but other individual projects for their own 
> jobs). Again, there'll be a goal champion to monitor and follow up.
> 
> 
> [1] 
> https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation-timeline.html
> [2] 
> https://governance.openstack.org/tc/reference/project-testing-interface.html#linux-distributions

Overall this approach seems fine. It is basically the approach we've used in the past with the addition of explicit testing of future python (which we've never really done before). I do think we need to avoid letting every project move at their own pace. This is what we did with the last python and distro switchover (trusty to xenial) and a number of projects failed to get it done within the release cycle. Instead I think we should rely on shared branch specific zuul templates that can be updated centrally when the next release is cut.

Clark