Open Stack

(I wrote the following response earlier this morning and then forgot to
hit send - sending just for the heck of it - I think Jim's response is
pretty complete)

On 08/09/2013 08:50 AM, Bob Ball wrote:
> Over the last few weeks I've been talking to Dan Prince about what we
> might be able to do with SmokeStack and whether we can get it to a
> stage where it can veto a change that it knows has broken tests.
> 
> The first part of this would need to be SmokeStack differentiating
> between the types of failure - this should be relatively easy and is
> planned in the next few weeks.
> 
> What we'd like to see once we have implemented this is a workflow
> similar to
> https://docs.google.com/presentation/d/1qTvZ8kmByOrtilYhc_BiwOhkzpN-GT0UTbniusnKohI/edit?usp=sharing,
> with appropriate integration with the gate.

The workflow seems sensible...

> What are the thoughts on how an external system can add gating
> comments?

We're generally very uncomfortable giving binding voting ability to any
systems that aren't run by the infra team - the main reason being that
if the gate breaks, we need to be able to fix it.

I'd start out with your below plan with automatic -1 votes and see how
it goes.

> My initial thought is that as soon as SmokeStack picks up a job it
> could -2 comment, preventing any merge.  Then it will be responsible
> for either a final +2 comment if the tests pass, -2 comment if a test
> explicitly fails and a 0 comment if a packaging failure or unknown
> failure occurs.
> 
> In the general case this should work fine because SmokeStack is
> running at sub 30 minutes per change, so Jenkins (typically taking
> around an hour) should complete well after SmokeStack and therefore
> would do the right thing without any further changes.
> 
> What are the thoughts on the case where SmokeStack takes longer to
> respond than Jenkins?
> 
> Some options might be: * Merge change after SmokeStack has approved
> (small risk that the merge might fail if SmokeStack approved changes
> in a different order) * Re-submit to the merge jobs after Smokestack
> has completed (ultra-safe, but with a double test for the happy
> path) * Merge anyway and accept that SmokeStack might find things to
> be broken in this and other changes

I don't think these matter, because we're talking about check jobs. The
median time between patch upload and approval/merge tends to be 1-2 weeks.

> In the final "merge anyway" case, we may need some way for users to
> ignore_smokestack -2's as a previous change may have broken tests not
> checked by the Jenkins gate. We'd need SmokeStack to have a retest
> keyword it watches for as well - or possibly just act on the
> recheck/reverify commands currently used.

All of the logic around when to run things and re-run things and
when/how to vote/retest is all in zuul already. If you're going to start
doing additional work on smokestack, it would probably be more
beneficial to not try to solve those problems in smokestack and instead
to rework smokestack to be a zuul worker. We've been discussing
non-Jenkins workers for zuul, now that it's on gearman - so maybe
smokestack would be an interesting exploration in that direction.

On the other hand, I would be remiss if I did not mention that I think
that engineering effort here could be better spent in improving the
gating jobs.

> Thanks,
> 
> Bob
> 
> _______________________________________________ OpenStack-Infra
> mailing list OpenStack-Infra at lists.openstack.org 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>