Open Stack

On Fri, Oct 19, 2018 at 3:46 PM Zane Bitter <zbitter at redhat.com> wrote:

> On 19/10/18 12:30 PM, Clark Boylan wrote:
> > On Fri, Oct 19, 2018, at 8:17 AM, Zane Bitter wrote:
> >> Unit Tests
> >> ----------
> >>
> >> For unit tests, the most important thing is to test on the versions of
> >> Python we target. It's less important to be using the exact distro that
> >> we want to target, because unit tests generally won't interact with
> >> stuff outside of Python.
> >>
> >> I'd like to propose that we handle this by setting up a unit test
> >> template in openstack-zuul-jobs for each release. So for Stein we'd have
> >> openstack-python3-stein-jobs. This template would contain:
> >
> > Because zuul config is branch specific we could set up every project to
> use a `openstack-python3-jobs` template then define that template
> differently on each branch. This would mean you only have to update the
> location where the template is defined and not need to update every other
> project after cutting a stable branch. I would suggest we take advantage of
> that to reduce churn.
>
> There was a reason I didn't propose that approach: in practice you can't
> add a new gating test to a centralised zuul template definition. If you
> do, many projects will break because the change is not self-testing. At
> best you'll be pitchforked by an angry mob of people who can't get
> anything but py37 fixes through the gate, and at worst they'll all stop
> using the template to get unblocked and then never go back to it.
>
> We don't need everyone to cut over at the same time. We just need them
> to do it in the space of one release cycle. One patch every 6 months is
> not an excessive amount of churn.
>
> >> * A voting gate job for the highest minor version of py3 we want to
> >> support in that release.
> >> * A voting gate job for the lowest minor version of py3 we want to
> >> support in that release.
> >> * A periodic job for any interim minor releases.
> >> * (Starting late in the cycle) a non-voting check job for the highest
> >> minor version of py3 we want to support in the *next* release (if
> >> different), on the master branch only.
> >>
> >> So, for example, (and this is still under active debate) for Stein we
> >> might have gating jobs for py35 and py37, with a periodic job for py36.
> >> The T jobs might only have voting py36 and py37 jobs, but late in the T
> >> cycle we might add a non-voting py38 job on master so that people who
> >> haven't switched to the U template yet can see what, if anything,
> >> they'll need to fix.
> >>
> >> We'll run the unit tests on any distro we can find that supports the
> >> version of Python we want. It could be a non-LTS Ubuntu, Fedora, Debian
> >> unstable, whatever it takes. We won't wait for an LTS Ubuntu to have a
> >> particular Python version before trying to test it.
> >>
> >> Before the start of each cycle, the TC would determine which range of
> >> versions we want to support, on the basis of the latest one we can find
> >> in any distro and the earliest one we're likely to need in one of the
> >> supported Linux distros. There will be a project-wide goal to switch the
> >> testing template from e.g. openstack-python3-stein-jobs to
> >> openstack-python3-treasure-jobs for every repo before the end of the
> >> cycle. We'll have goal champions as usual following up and helping teams
> >> with the process. We'll know where the problem areas are because we'll
> >> have added non-voting jobs for any new Python versions to the previous
> >> release's template.
> >
> > I don't know that this needs to be a project wide goal if you can just
> update the template on the master branch where the template is defined. Do
> that then every project is now running with the up to date version of the
> template. We should probably advertise when this is happening with some
> links to python version x.y breakages/features, but the process itself
> should be quick.
>
> Either way, it'll be project teams themselves fixing any broken tests
> due to a new version being added. So we can either have a formal
> project-wide goal where we project-manage that process across the space
> of a release, or a de-facto project-wide goal where we break everybody
> and then nothing gets merged until they fix it.
>
> > As for python version range selection I worry that that the criteria
> about relies on too much guesswork.
>
> Some guesswork is going to be inevitable, unfortunately, (we have no way
> of knowing what will be in CentOS 8, for example) but I agree that we
> should try to tighten up the criteria as much as possible.
>
> > I do think we should do our best to test future incoming versions of
> python even while not officially supporting them. We will have to support
> them at some point, either directly or via some later version that includes
> the changes from that intermediate version.
>
> +1, I think we should try to add support for higher versions as soon as
> possible. It may take a long time to get into an LTS release, but
> there's bound to be _some_ distro out there where people want to use it.
> (Case in point: Debian really wanted py37 support in Rocky, at which
> point a working 3.7 wasn't even available in _any_ Ubuntu release, let
> alone an LTS). That's why I said "the latest one we can find in any
> distro" - if we have any way to test it at all then we should.
>
> > Could the criteria be:
> > Support the lowest version of python on a supported distro release and
> the highest version of python on a supported distro
>
> As of now we can't objectively determine the minimum version because it
> isn't the future yet. That may change once every distro is on Python 3
> though.
>
> > and test (but not support so we can drop testing for this python version
> on stable branches) the current latest release of python?
>
> That's certainly worth considering; we could tighten up the range on
> stable branches. I think we'd need to hear from Ubuntu and Debian folks
> what they think about that. My guess is that they'd prefer even stable
> branches to continue testing recent versions, so that they could be used
> on Debian unstable and on Ubuntu between LTS releases.
>
>
Zane and all, thanks very much for pushing this initiative forward. The
general approach makes sense to me. Proactively testing with the latest
python versions would be awesome and would close a much needed gap.

As for testing stable branches with the latest Python versions. I believe
this refers to the latest Python version available anywhere, ie. perhaps an
unstable release from sid. In general it would make sense to to continue
testing the most recent stable release the same way it was tested when it
was master. There's a good chance that by the time stable/xyz branches are
cut from master, the unstable Python version that was being tested in
master is now in a stable distro release. For example, for the OpenStack U
release this will most likely be the case for Ubuntu with Python 3.8.

Here are the versions we support in Ubuntu, and (most likely) will support
in the future, in case it's helpful:

Queens: py3.5 (16.04); py3.6 (18.04)
Rocky: py3.6 (18.04); py3.6 (18.10)
Stein: py3.6 (18.04); py3.7 (19.04)
T: py3.6 (18.04); py3.7 (19.10)
U: py3.6 (18.04); py3.8 (20.04)
V: py3.8 (20.04); py3.? (20.10)

Thanks,
Corey


> > This is objective and doesn't require anyone to guess at what versions
> need to be supported.
> >
> >>
> >> Integration Tests
> >> -----------------
> >>
> >> Integration tests do test, amongst other things, integration with
> >> non-openstack-supplied things in the distro, so it's important that we
> >> test on the actual distros we have identified as popular.[2] It's also
> >> important that every project be testing on the same distro at the end of
> >> a release, so we can be sure they all work together for users.
> >>
> >> When a new release of CentOS or a new LTS release of Ubuntu comes out,
> >> the TC will create a project-wide goal for the *next* release cycle to
> >> switch all integration tests over to that distro. It's up to individual
> >> projects to make the switch for the tests that they own (e.g. it'd be
> >> the QA team for Tempest, but other individual projects for their own
> >> jobs). Again, there'll be a goal champion to monitor and follow up.
> >>
> >>
> >> [1]
> >>
> https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation-timeline.html
> >> [2]
> >>
> https://governance.openstack.org/tc/reference/project-testing-interface.html#linux-distributions
> >
> > Overall this approach seems fine. It is basically the approach we've
> used in the past with the addition of explicit testing of future python
> (which we've never really done before). I do think we need to avoid letting
> every project move at their own pace. This is what we did with the last
> python and distro switchover (trusty to xenial) and a number of projects
> failed to get it done within the release cycle. Instead I think we should
> rely on shared branch specific zuul templates that can be updated centrally
> when the next release is cut.
>
> For those just joining, we discussed this on IRC yesterday.[1] fungi
> mentioned that we tried two different approaches for precise->trusty and
> trusty->xenial, and they both failed in different ways. The first time
> infra gave teams time to prepare and then eventually cut the build over
> for everyone, with the result that lots of things broke. The second time
> infra allowed teams to switch over in their own time, with the result
> that a lot of things released with tests running on an outdated LTS
> release.
>
> We do have one new tool in our toolbox: project-wide goals. They're not
> magically going to solve the problem, but at least they give us some
> visibility into what has and has not happened. Perhaps we could provide
> periodic, rather than experimental, jobs to test with so that the goal
> champions can track where the likely problems are in the lead-up to a
> switchover? As far as the LTS distro goes, I don't have a strong opinion
> on which approach is the least worst. I think if we want to have a
> switchover (just after milestone-2 maybe?) then we should just write
> that into the project-wide goal.
>
> cheers,
> Zane.
>
> [1]
>
> http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2018-10-18.log.html#t2018-10-18T15:16:13
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20181022/e1a3d1a5/attachment.html>