Open Stack

On 10/28/2014 02:15 PM, Aleksandra Fedorova wrote:
> Vitaly,
>
> though comments like this are definitely better than nothing, I think
> we should address these issues in a more formal way.
>
> For random failures we have to retrigger the build until it passes.
> Yes, it could take some time (two-three rebuilds?), but it is the only
> reliable way which shows that it is indeed random and hasn't suddenly
> become permanent. If it fails three times in a row, this issue is
> probably bigger than you think. Should we really ignore/postpone it
> then?
>
> And if it is really the known issue, we need to fix or disable this
> particular test. And I think that this fix should be merged in the
> repo via the general workflow.
>
> It doesn't only make everything pass the CI properly, it also adds
> this necessary step where you announce the issue publicly and it gets
> approved as the "official" known issue. I would even add certain
> keyword for the commit message to mark this temporary fixes to
> simplify tracking.

Aleksandra,

You are 100% correct here. Under no circumstances should any human be 
able to merge code into a master source tree. Only the CI system, after 
a successful run of tests, should be able to merge code into master. If 
there are, as Vitaly says, issues with a nailgun test that cause random 
failures, then the test (or nailgun, whichever is the cause) should be 
fixed ASAP.

We deal with similar issues in the main OpenStack gate, and luckily 
there we don't allow humans to merge code directly into a branch. Only 
the CI system can do that, which means that although at times we get 
frustrated developers who must "do the recheck dance" a bit, there is a 
forcing function to have developers fix bugs in tests and server code 
that trigger false failures.

All the best, and keep up the good work.

-jay