Open Stack

I realised I hadn't replied to this one!

> -----Original Message-----
> Just so I understand the suggestion, you're saying you can make
> smokestack test the changes stacked in the same sequence they're
> being started by zuul, watch zuul's status mirroring the various
> change ejections/test restarts... and finish testing on a given set
> of scenarios before zuul does?

Many of the changes included by Zuul aren't tested, or don't need to be tested by SmokeStack - the obvious example being devstack, but there are several other projects that are tested by zuul that SS doesn't track.

The changes are also primarily tested in isolation - so not in the strict ordering included in the gate.  I know that with SmokeStack not following this strict ordering there is a risk that a false pass will slip through - but false fails shouldn't occur because SmokeStack does also consider the dependencies that determine the gate ordering.

As such, my view is that SmokeStack could give a useful -2 vote when it knows a patch is broken - and yes, these tend to (although not 100%) complete before the gating jobs have completed for a change.

> You had mentioned something about smokestack adding a -2 when it
> begins testing and removing it on success, but if it doesn't
> complete before zuul is ready to merge those commits then how do we
> differentiate between smokestack still running (so we know to pause
> the queue) and smokestack leaving that -2 indefinitely because the
> test failed? Also worth noting, gerrit doesn't clear a -2 on a new
> patchset upload like it does with a -1 or a positive vote, so the
> proposed solution would additionally need to take care of rescinding
> those -2 votes itself when new patchsets arrive to correct the
> failure.

I hadn't understood the implications properly - and I agree now that posting a -2 at start-of-tests would not be a good idea.

My suggestion now is, in a nutshell, to allow SmokeStack to post a -2 review on a gating job so that, if the tests fail (i.e. not packaging) and it completes before the gate, then Smokestack can prevent a broken change being merged.

Is that something which could work?

Bob