Open Stack

On 07/13/2018 06:38 AM, Thomas Goirand wrote:
> Now, both Debian and Ubuntu have Python 3.7. Every package which I
> upload in Sid need to support that. Yet, OpenStack's CI is still
> lagging with Python 3.5.

OpenStack's CI is rather broad -- I'm going to assume we're talking
about whole-system devstack-ish based functional tests.  Yes most
testing is on Xenial and hence Python 3.5

We have Python 3.6 available via Bionic nodes.  I think current talk
is to look at mass-updates after the next release.  Such updates, from
history, are fairly disruptive.

> I'm aware that there's been some attempts in the OpenStack infra to
> have Debian Sid (which is probably the distribution getting the
> updates the faster).

We do not currently build Debian sid images, or mirror the unstable
repos or do wheel builds for Debian.  diskimage-builder also doesn't
test it in CI.  This is not to say it can't be done.

> If it cannot happen with Sid, then I don't know, choose another
> platform, and do the Python 3-latest gating...

Fedora has been consistently updated in OpenStack Infra for many
years.  IMO, and from my experience, six-monthly-ish updates are about
as frequent as can be practically handled.

The ideal is that a (say) Neutron dev gets a clear traceback from a
standard Python error in their change and happily fixes it.  The
reality is probably more like this developer gets a tempest
failure due to nova failing to boot a cirros image, stemming from a
detached volume due to a qemu bug that manifests due to a libvirt
update (I'm exaggerating, I know :).

That sort of deeply tangled platform issue always exists; however it
is armortised across the lifespan of the testing.  So several weeks
after we update all these key components, a random Neutron dev can be
pretty sure that submitting their change is actually testing *their*
change, and not really a defacto test of every other tangentially
related component.

A small, but real example; uwsgi wouldn't build with the gcc/glibc
combo on Fedora 28 for two months after its release until uwsgi's
2.0.17.1.  Fedora carried patches; but of course there were a lot
previously unconsidered assumptions in devstack around deployment that
made using the packaged versions difficult [1] (that stack still
hasn't received any reviews).

Nobody would claim diskimage-builder is the greatest thing ever, but
it does produce our customised images in a wide variety of formats
that runs in our very heterogeneous clouds.  It's very reactive -- we
don't know about package updates until they hit the distro, and
sometimes that breaks assumptions.  It's largely taken for granted in
our CI, but it takes a constant sustained effort across the infra team
to make sure we have somewhere to test.

I hear myself sounding negative, but I think it's a fundamental
problem.  You can't be dragging in the latest of everything AND expect
that you won't be constantly running off fixing weird things you never
even knew existed.  We can (and do) get to the bottom of these things,
but if the platform changes again before you've even fixed the current
issue, things start piling up.

If the job is constantly broken it gets ignored -- if a non-voting
job fails in the woods, does it make a sound? :)

> When this happens, moving faster with Python 3 versions will be
> mandatory for everyone, not only for fools like me who made the
> switch early.

This is a long way of saying that - IMO - the idea of putting out a
Debian sid image daily (to a lesser degree Buster images) and throwing
a project's devstack runs against it is unlikely to produce a good
problems-avoided : development-resources ratio.  However, prove me
wrong :)

If people would like to run their master against Fedora (note
OpenStack's stable branch lifespan is generally longer than a given
Fedora release is supported, so it is not much good there) you have
later packages, but still a fairly practical 6-month-ish stability
cadence.  I'm happy to help (some projects do already).

> </end of ranting>

With my rant done :) ... there's already discussion around multiple
python versions, containers, etc in [2].  While I'm reserved about the
idea of full platform functional tests, essentially having a
wide-variety of up-to-date tox environments using some of the methods
discussed there is, I think, a very practical way to be cow-catching
some of the bigger issues with Python version updates.  If we are to
expend resources, my 2c worth is that pushing in that direction gives
the best return on effort.

-i

[1] https://review.openstack.org/#/c/565923/
[2] http://lists.openstack.org/pipermail/openstack-dev/2018-July/132152.html