Open Stack

On 08/14/2013 06:08 AM, Bob Ball wrote:
> I realised I hadn't replied to this one!
> 
>> -----Original Message----- Just so I understand the suggestion,
>> you're saying you can make smokestack test the changes stacked in
>> the same sequence they're being started by zuul, watch zuul's
>> status mirroring the various change ejections/test restarts... and
>> finish testing on a given set of scenarios before zuul does?
> 
> Many of the changes included by Zuul aren't tested, or don't need to
> be tested by SmokeStack - the obvious example being devstack, but
> there are several other projects that are tested by zuul that SS
> doesn't track.
> 
> The changes are also primarily tested in isolation - so not in the
> strict ordering included in the gate.  I know that with SmokeStack
> not following this strict ordering there is a risk that a false pass
> will slip through - but false fails shouldn't occur because
> SmokeStack does also consider the dependencies that determine the
> gate ordering.

If you aren't testing with the re-ordering, then you are doing the
equivalent of testing the patch as uploaded. (which, it turns out, is
super useful)

> As such, my view is that SmokeStack could give a useful -2 vote when
> it knows a patch is broken - and yes, these tend to (although not
> 100%) complete before the gating jobs have completed for a change.
> 
>> You had mentioned something about smokestack adding a -2 when it 
>> begins testing and removing it on success, but if it doesn't 
>> complete before zuul is ready to merge those commits then how do
>> we differentiate between smokestack still running (so we know to
>> pause the queue) and smokestack leaving that -2 indefinitely
>> because the test failed? Also worth noting, gerrit doesn't clear a
>> -2 on a new patchset upload like it does with a -1 or a positive
>> vote, so the proposed solution would additionally need to take care
>> of rescinding those -2 votes itself when new patchsets arrive to
>> correct the failure.
> 
> I hadn't understood the implications properly - and I agree now that
> posting a -2 at start-of-tests would not be a good idea.
> 
> My suggestion now is, in a nutshell, to allow SmokeStack to post a -2
> review on a gating job so that, if the tests fail (i.e. not
> packaging) and it completes before the gate, then Smokestack can
> prevent a broken change being merged.
> 
> Is that something which could work?

In practice, honestly, until I see it run with automatic -1's for a
while, I think we're stepping several steps beyond what we know.

I want to suggest that:

a) get auto -1 to a place where you're happy with it
b) after that - let's look at stats on incidences of -1 votes from
smokestack that were overridden erroneously.

I think both of the above are useful and non-controversial things. And
then once they're done, we'll have better data to base further
discussions on.