Open Stack

On 23 January 2014 09:39, Sean Dague <sean at dague.net> wrote:
> ================================
> Changes coming in gate structure
> ================================

> Svelt Gate
> ==========
>
> The gate jobs will be trimmed down immensely. Nothing project
> specific, so pep8 / unit tests all ripped out, no functional test
> runs. Less overall configs. Exactly how minimal we'll figure out as we
> decide what we can live without. The floor for this would be
> devstack-tempest-full and grenade.
>
> This is basically sanity check that the combination of patches in
> flight doesn't ruin the world for everyone.

So two things occur to me here -
 - this increases thread-the-needle risks.
 - what value does the sanity check still offer?

To illustrate this, consider that we have one patch for consideration.
We'll run it in check, then in gate, then it lands.

If a second patch is submitted, it will run in isolation in check, so
we have a cross-p*atch* race - e.g. a change(1) to nova won't be
tested with nova(2) until the gate run runs.

If the change is only applicable to one configured scenario, that
scenario has to be in the gate, or its not tested at all and we find
out that the thing is broken when *every nova check run from that
point on breaks* - instant firedrill.

The risk here seems to be roughly P(broken-in-combination) *
patches-not-tested-in-combination : e.g. if checks and gate are both
0.5h long [under estimated round number for ease of calculations] then
from clicking approve we have one hour until the change is in trunk
and we're immune to interactions being undetected. If we're pushing
500 through a day at peak, then in that interval 20 patches could be
submitted on average.

So for anything we take out of the gate and make check only our risk
of cross-patch needle getting through is up to twenty times higher
than it is of such a  thing breaking the gate today. OTOH as you say,
our gate breaks aren't typically in e.g. unit tests.

For things that are in the gate, obviously it's business as usual.

So - I'm pretty worried that we're going to have more firedrills with
this, in things like unittests and functional tests, even while
tempest scenarios are kept running.

-Rob


-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud