Open Stack

On Tue, Dec 01, 2015 at 06:00:26PM -0500, Zane Bitter wrote:
> On 01/12/15 06:22, Steven Hardy wrote:
> >>>      +1
> >>>
> >>>      I don't think it hurts to turn it on, but tbh I'm uncomfortable with the
> >>>      mental overhead of a non-voting job that I have to manually treat as a
> >>>      voting job. If it's stable enough to make it a voting job, I'd prefer we
> >>>      just make it voting. And if it's not then I'd like to see it be made
> >>>      stable enough to be a voting job and then make it voting.
> >>>
> >>>    This is roughly where I sit as well -- if it's non-voting, experience
> >>>    tells me that it will largely be ignored, and as such, isn't a good use of
> >>>    resources.
> >I'm sure you can appreciate it's something of a chicken/egg problem though
> >- if everyone always ignores non-voting jobs, they never become voting.
> >
> >That effect is magnified with TripleO though, because it consumes so many
> >OpenStack projects, any one of which has the capability to break our CI, so
> >in an ideal world we'd have voting feedback on all-the-things, but that's
> >not where we are right now due in large-part to the steady stream of
> >regressions (from Heat, Ironic and other projects).
> 
> Yeah, but when it does break the problem becomes that reviewers now have to
> debug TripleO tests on every single patchset to see if they broke it (or
> made it worse) or not. Pretty soon you never know whether you should be
> ignoring it or not, and... we all know how this story ends.

Well, I see your point, but it's not that hard - if every single TripleO
job has failed on every single revision of your patch (assuming there is
more than one), and you've checked TripleO CI isn't broken, your patch is
almost certainly to blame.

> Maybe it would help if there was a periodic job running TripleO tests
> against master and posting the results somewhere. If I could go to
> aretripleocheckfailuresreal.com and it just says YES or NO in giant 8-inch
> high letters, that would help a lot. The test could run let's say once an
> hour and if the last 3 all passed then we'd call that YES.
> 
> Even better would be if we could tie this into the gate, so that it runs as
> a gating test when the tests are working and not when they aren't. I don't
> think infra will thank me for suggesting that though ;)

Neither of those approaches strike me as particularly workable at least in
the short term.  Hence the simpler suggestion that folks just watch out for
a consistent series of red feedback and think twice before +Aing when they see
it.

> It also occurs to me that the trade-off may be different for stable
> branches. It might well be possible to start gating stable/liberty Heat
> against TripleO right away.

Yeah, I think we could potentially be much stricter with stable branches
because the velocity is much lower.

Cheers,

Steve