<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Aug 17, 2013 at 9:46 AM, Robert Collins <span dir="ltr"><<a href="mailto:robertc@robertcollins.net" target="_blank">robertc@robertcollins.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im">On 17 August 2013 23:49, Salvatore Orlando <<a href="mailto:sorlando@nicira.com">sorlando@nicira.com</a>> wrote:<br>
> I tend to agree that when the gate for a project is broken, nothing should<br>
> be merged for that project until the gate jobs are green again.<br>
> In the case of Neutron, making the job non voting only caused more bugs to<br>
> slip through, and that meant more works for the developer themselves, and<br>
> more headaches for developers of other projects relying on it.<br>
<br>
<br>
<br>
> When dealing with intermittent failures, like the bug which probably started<br>
> the issues we've been witnessing in the past 3 weeks, I think it might a<br>
> sensible idea to make the job non-voting only for projects which surely<br>
> can't be the cause of the gate failure; or perhaps skip the offending test<br>
> only.<br>
><br>
> This means however asymettrical gating, and from Monty's post it seems<br>
> there's something quite wrong with it. However, due to my lack of expertise<br>
> on the subject, I am unable to see the issue with it.<br>
><br>
> Salvatore<br>
<br>
</div>The asymmetry we should fear is when project A can land something<br>
something which will break project B. In this case the proposal is to<br>
say 'B is broken already, permit A to land things without remorse<br>
until B is unbroken'.<br>
<br>
The problem is, if A makes the breakage of B worse, B ends up in<br>
catchup mode, which is most unfun.<br>
<br>
Concretely, take heat for A and neutron for B. Tempest d-g jobs start<br>
failing in neutron, so they are made skips. Now heat could make<br>
neutron tests in tempest worse, and we won't know - or if we do know,<br>
they'll still land.<br>
<br>
Previous discussion here has endorsed 'revert problematic commits,<br>
it's not blame on the developer, just do it', so I'm not going to<br>
mention that.<br>
<br>
What I will suggest we do is start running some number - lets say 20 -<br>
of midnight state jobs, all identical. Ignoring datetime sensitive<br>
tests, which are fortunately rare, this should identify tests that<br>
fail 5% of the time, independent of incoming commits. We can use this<br>
to generate a baseline reference for which tests fail intermittently<br>
in trunk, and when something breaks intermittently outside of that<br>
set, we can be pretty *sure* it's in the last days commits.<br></blockquote><div><br></div><div>+1, although we already have a manual vaguely similar version of this (<a href="http://status.openstack.org/rechecks/">http://status.openstack.org/rechecks/</a>)</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Secondly, in principle it should be straight forward to do this for<br>
any point in time, so when a new problem shows it's head, we can start<br>
a bisection up programmatically - independent of the dev analysis - to<br>
find where it was introduced. If we have resources we could even do<br>
N-section rather than bisection.</blockquote><div><br></div><div>+1</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Killing all intermittent issues test suites is /hard/, so I think we<br>
need to have a belt-and-braces approach and engineer a rapid response<br>
system to spikes in intermittent failures, in addition to working on<br>
the failures themselves. </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
-Rob<br>
<span class=""><font color="#888888">--<br>
Robert Collins <<a href="mailto:rbtcollins@hp.com">rbtcollins@hp.com</a>><br>
Distinguished Technologist<br>
HP Converged Cloud<br>
</font></span><div class=""><div class="h5"><br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>