Open Stack

On 01/12/15 06:22, Steven Hardy wrote:
>> >      +1
>> >
>> >      I don't think it hurts to turn it on, but tbh I'm uncomfortable with the
>> >      mental overhead of a non-voting job that I have to manually treat as a
>> >      voting job. If it's stable enough to make it a voting job, I'd prefer we
>> >      just make it voting. And if it's not then I'd like to see it be made
>> >      stable enough to be a voting job and then make it voting.
>> >
>> >    This is roughly where I sit as well -- if it's non-voting, experience
>> >    tells me that it will largely be ignored, and as such, isn't a good use of
>> >    resources.
> I'm sure you can appreciate it's something of a chicken/egg problem though
> - if everyone always ignores non-voting jobs, they never become voting.
>
> That effect is magnified with TripleO though, because it consumes so many
> OpenStack projects, any one of which has the capability to break our CI, so
> in an ideal world we'd have voting feedback on all-the-things, but that's
> not where we are right now due in large-part to the steady stream of
> regressions (from Heat, Ironic and other projects).

Yeah, but when it does break the problem becomes that reviewers now have 
to debug TripleO tests on every single patchset to see if they broke it 
(or made it worse) or not. Pretty soon you never know whether you should 
be ignoring it or not, and... we all know how this story ends.

Maybe it would help if there was a periodic job running TripleO tests 
against master and posting the results somewhere. If I could go to 
aretripleocheckfailuresreal.com and it just says YES or NO in giant 
8-inch high letters, that would help a lot. The test could run let's say 
once an hour and if the last 3 all passed then we'd call that YES.

Even better would be if we could tie this into the gate, so that it runs 
as a gating test when the tests are working and not when they aren't. I 
don't think infra will thank me for suggesting that though ;)


It also occurs to me that the trade-off may be different for stable 
branches. It might well be possible to start gating stable/liberty Heat 
against TripleO right away.

cheers,
Zane.