[openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

Mike Bayer mbayer at redhat.com
Wed Dec 16 21:50:39 UTC 2015

On 12/16/2015 01:32 PM, Jeremy Stanley wrote:
> On 2015-12-16 11:03:38 -0700 (-0700), Carl Baldwin wrote:
> [...]
>> We need to vet new package releases before they wreak havoc.  We need
>> to accept new package releases by proposing a patch to update the
>> version and take it through the gate.  Weren't we working on this at
>> one point?  I understand if it isn't quite possible to do this yet but
>> we need to be working toward this and accelerating our efforts rather
>> than lashing out at package maintainers.
> [...]
> Yes, it's progressing nicely. DevStack-based jobs are already
> covered this way for master and stable/liberty, and Neutron is
> piloting the same solution for its other non-DevStack-based jobs. If
> Nova's unit test jobs were already switched to their
> upper-constraints equivalents then there's a chance this wouldn't
> have impacted there (though we still need to work out the bit where
> we run a representative sample of jobs like neutron/nova unit tests
> on proposed constraints bumps to block any with this sort of impact,
> right now we're really just relying on
> devstack-tempest/grenade/other integration test jobs as canaries).
> Anyway, the solution seems to be working (modulo unforeseen issues
> like people thinking it's sane to delete their latest releases of
> some dependencies from PyPI) but it's a long road to full
> implementation.

So just FTR I mistakenly thought that the upper-constraints system was
in place for all gate jobs, and that the release of Alembic 0.8.4 would
at worst have raised a message somewhere within the system that updates
the upper-constraints file, and prevented the version bump from
proceeding.  Had I known that the raw master gate jobs for Nova were
going to go down overnight, I would not have released Alembic in this way.

That said, as the upper-constraints system is available, and not using
it means that any upstream package release that breaks any test will
cause 12 hours of gate downtime, I'm surprised that rolling this system
out across the board isn't an emergency priority.     Because right now,
literally any of the 50 or so projects I see in Nova requirements.txt
can release something on Pypi at any moment and cause another 12 hours
of downtime.   Lots of them don't even have major version upper bounds,
even complex and intricate dependencies like lxml and boto.   If a
single test breakage due to an incompatible change in an upstream
release means 100 man hours lost and 12 hours of downtime, then Nova/
others are essentially a giant boulder balanced on the edge of a cliff,
with 50 or so loaded BB guns on Pypi pointed at it.


More information about the OpenStack-dev mailing list