Open Stack

On 19/10/18 12:30 PM, Clark Boylan wrote:
> On Fri, Oct 19, 2018, at 8:17 AM, Zane Bitter wrote:
>> Unit Tests
>> ----------
>>
>> For unit tests, the most important thing is to test on the versions of
>> Python we target. It's less important to be using the exact distro that
>> we want to target, because unit tests generally won't interact with
>> stuff outside of Python.
>>
>> I'd like to propose that we handle this by setting up a unit test
>> template in openstack-zuul-jobs for each release. So for Stein we'd have
>> openstack-python3-stein-jobs. This template would contain:
> 
> Because zuul config is branch specific we could set up every project to use a `openstack-python3-jobs` template then define that template differently on each branch. This would mean you only have to update the location where the template is defined and not need to update every other project after cutting a stable branch. I would suggest we take advantage of that to reduce churn.

There was a reason I didn't propose that approach: in practice you can't 
add a new gating test to a centralised zuul template definition. If you 
do, many projects will break because the change is not self-testing. At 
best you'll be pitchforked by an angry mob of people who can't get 
anything but py37 fixes through the gate, and at worst they'll all stop 
using the template to get unblocked and then never go back to it.

We don't need everyone to cut over at the same time. We just need them 
to do it in the space of one release cycle. One patch every 6 months is 
not an excessive amount of churn.

>> * A voting gate job for the highest minor version of py3 we want to
>> support in that release.
>> * A voting gate job for the lowest minor version of py3 we want to
>> support in that release.
>> * A periodic job for any interim minor releases.
>> * (Starting late in the cycle) a non-voting check job for the highest
>> minor version of py3 we want to support in the *next* release (if
>> different), on the master branch only.
>>
>> So, for example, (and this is still under active debate) for Stein we
>> might have gating jobs for py35 and py37, with a periodic job for py36.
>> The T jobs might only have voting py36 and py37 jobs, but late in the T
>> cycle we might add a non-voting py38 job on master so that people who
>> haven't switched to the U template yet can see what, if anything,
>> they'll need to fix.
>>
>> We'll run the unit tests on any distro we can find that supports the
>> version of Python we want. It could be a non-LTS Ubuntu, Fedora, Debian
>> unstable, whatever it takes. We won't wait for an LTS Ubuntu to have a
>> particular Python version before trying to test it.
>>
>> Before the start of each cycle, the TC would determine which range of
>> versions we want to support, on the basis of the latest one we can find
>> in any distro and the earliest one we're likely to need in one of the
>> supported Linux distros. There will be a project-wide goal to switch the
>> testing template from e.g. openstack-python3-stein-jobs to
>> openstack-python3-treasure-jobs for every repo before the end of the
>> cycle. We'll have goal champions as usual following up and helping teams
>> with the process. We'll know where the problem areas are because we'll
>> have added non-voting jobs for any new Python versions to the previous
>> release's template.
> 
> I don't know that this needs to be a project wide goal if you can just update the template on the master branch where the template is defined. Do that then every project is now running with the up to date version of the template. We should probably advertise when this is happening with some links to python version x.y breakages/features, but the process itself should be quick.

Either way, it'll be project teams themselves fixing any broken tests 
due to a new version being added. So we can either have a formal 
project-wide goal where we project-manage that process across the space 
of a release, or a de-facto project-wide goal where we break everybody 
and then nothing gets merged until they fix it.

> As for python version range selection I worry that that the criteria about relies on too much guesswork.

Some guesswork is going to be inevitable, unfortunately, (we have no way 
of knowing what will be in CentOS 8, for example) but I agree that we 
should try to tighten up the criteria as much as possible.

> I do think we should do our best to test future incoming versions of python even while not officially supporting them. We will have to support them at some point, either directly or via some later version that includes the changes from that intermediate version.

+1, I think we should try to add support for higher versions as soon as 
possible. It may take a long time to get into an LTS release, but 
there's bound to be _some_ distro out there where people want to use it. 
(Case in point: Debian really wanted py37 support in Rocky, at which 
point a working 3.7 wasn't even available in _any_ Ubuntu release, let 
alone an LTS). That's why I said "the latest one we can find in any 
distro" - if we have any way to test it at all then we should.

> Could the criteria be:
> Support the lowest version of python on a supported distro release and the highest version of python on a supported distro

As of now we can't objectively determine the minimum version because it 
isn't the future yet. That may change once every distro is on Python 3 
though.

> and test (but not support so we can drop testing for this python version on stable branches) the current latest release of python?

That's certainly worth considering; we could tighten up the range on 
stable branches. I think we'd need to hear from Ubuntu and Debian folks 
what they think about that. My guess is that they'd prefer even stable 
branches to continue testing recent versions, so that they could be used 
on Debian unstable and on Ubuntu between LTS releases.

> This is objective and doesn't require anyone to guess at what versions need to be supported.
> 
>>
>> Integration Tests
>> -----------------
>>
>> Integration tests do test, amongst other things, integration with
>> non-openstack-supplied things in the distro, so it's important that we
>> test on the actual distros we have identified as popular.[2] It's also
>> important that every project be testing on the same distro at the end of
>> a release, so we can be sure they all work together for users.
>>
>> When a new release of CentOS or a new LTS release of Ubuntu comes out,
>> the TC will create a project-wide goal for the *next* release cycle to
>> switch all integration tests over to that distro. It's up to individual
>> projects to make the switch for the tests that they own (e.g. it'd be
>> the QA team for Tempest, but other individual projects for their own
>> jobs). Again, there'll be a goal champion to monitor and follow up.
>>
>>
>> [1]
>> https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation-timeline.html
>> [2]
>> https://governance.openstack.org/tc/reference/project-testing-interface.html#linux-distributions
> 
> Overall this approach seems fine. It is basically the approach we've used in the past with the addition of explicit testing of future python (which we've never really done before). I do think we need to avoid letting every project move at their own pace. This is what we did with the last python and distro switchover (trusty to xenial) and a number of projects failed to get it done within the release cycle. Instead I think we should rely on shared branch specific zuul templates that can be updated centrally when the next release is cut.

For those just joining, we discussed this on IRC yesterday.[1] fungi 
mentioned that we tried two different approaches for precise->trusty and 
trusty->xenial, and they both failed in different ways. The first time 
infra gave teams time to prepare and then eventually cut the build over 
for everyone, with the result that lots of things broke. The second time 
infra allowed teams to switch over in their own time, with the result 
that a lot of things released with tests running on an outdated LTS release.

We do have one new tool in our toolbox: project-wide goals. They're not 
magically going to solve the problem, but at least they give us some 
visibility into what has and has not happened. Perhaps we could provide 
periodic, rather than experimental, jobs to test with so that the goal 
champions can track where the likely problems are in the lead-up to a 
switchover? As far as the LTS distro goes, I don't have a strong opinion 
on which approach is the least worst. I think if we want to have a 
switchover (just after milestone-2 maybe?) then we should just write 
that into the project-wide goal.

cheers,
Zane.

[1] 
http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2018-10-18.log.html#t2018-10-18T15:16:13