Open Stack

Robert Collins <robertc at robertcollins.net> writes:

> On 23 January 2014 09:39, Sean Dague <sean at dague.net> wrote:
>> ================================
>> Changes coming in gate structure
>> ================================
>
>> Svelt Gate
>> ==========
>>
>> The gate jobs will be trimmed down immensely. Nothing project
>> specific, so pep8 / unit tests all ripped out, no functional test
>> runs. Less overall configs. Exactly how minimal we'll figure out as we
>> decide what we can live without. The floor for this would be
>> devstack-tempest-full and grenade.
>>
>> This is basically sanity check that the combination of patches in
>> flight doesn't ruin the world for everyone.
>
> So two things occur to me here -
>  - this increases thread-the-needle risks.
>  - what value does the sanity check still offer?

Here's how I see this: we have a process that ensures that the code
remains as perfect as its tests.  We have seen that the combined system
of openstack-and-its-tests is not perfect.  The complexity of that
system is great, and it seems we will likely never remove all
non-deterministic failures from this system.  At least, we seem to be
able to introduce them as fast as they are removed.

So this is an adaptation to two aspects of our current situation:
single-project-test failures are causing gate failures, and the
non-deterministic failure rate in cross-project-tests are as well.  The
first is addressed by moving single-project tests out of the gate.  So a
single project is more likely to break itself but less likely to break
others.  This is a trade-off, but it localizes the pain in case of
error.  The second is addressed by removing some of our
cross-project-test variants from the gate.  The more variants we run,
the more likely we are to hit non-deterministic bugs.  You have argued
before that rate is not high enough for us to prevent these bugs from
entering, but perhaps with the reduction it will be low enough that it
allows us to get work done.

It's possible both of those decisions may let in more bugs than we are
comfortable with.  However, given the current state where cross-project
frustrations are high and development has slowed dramatically,
localizing failures when possible and focusing the gate on ensuring that
there's at least a basic workable system seems worth trying.

If we take too much out of the gate, we can put it back in.  Regardless,
I think the changes to require recent check votes will help and be a
long-lasting improvement.

-Jim