<div dir="ltr">Overall this looks really good, and very spot on.<div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jan 2, 2014 at 6:29 PM, Sean Dague <span dir="ltr"><<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">A lot of elastic recheck this fall has been based on the ad hoc needs of the moment, in between diving down into the race bugs that were uncovered by it. This week away from it all helped provide a little perspective on what I think we need to do to call it *done* (i.e. something akin to a 1.0 even though we are CDing it).<br>
<br>
Here is my current thinking on the next major things that should happen. Opinions welcomed.<br>
<br>
(These are roughly in implementation order based on urgency)<br>
<br>
= Split of web UI =<br>
<br>
The elastic recheck page is becoming a mismash of what was needed at the time. I think what we really have emerging is:<br>
* Overall Gate Health<br>
* Known (to ER) Bugs<br>
* Unknown (to ER) Bugs - more below<br>
<br>
I think the landing page should be Know Bugs, as that's where we want both bug hunters to go to prioritize things, as well as where people looking for known bugs should start.<br>
<br>
I think the overall Gate Health graphs should move to the zuul status page. Possibly as part of the collection of graphs at the bottom.<br>
<br>
We should have a secondary page (maybe column?) of the un-fingerprinted recheck bugs, largely to use as candidates for fingerprinting. This will let us eventually take over /recheck.<br>
<br></blockquote><div><br></div><div>I think it would be cool to collect the list of unclassified failures (not by recheck bug), so we can see how many (and what percentage) need to be classified. This isn't gate health but more of e-r health or something like that.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
= Data Analysis / Graphs =<br>
<br>
I spent a bunch of time playing with pandas over break (<a href="http://dague.net/2013/12/30/ipython-notebook-experiments/" target="_blank">http://dague.net/2013/12/30/<u></u>ipython-notebook-experiments/</a>)<u></u>, it's kind of awesome. It also made me rethink our approach to handling the data.<br>
<br>
I think the rolling average approach we were taking is more precise than accurate. As these are statistical events they really need error bars. Because when we have a quiet night, and 1 job fails at 6am in the morning, the 100% failure rate it reflects in grenade needs to be quantified that it was 1 of 1, not 50 of 50.</blockquote>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
So my feeling is we should move away from the point graphs we have, and present these as weekly and daily failure rates (with graphs and error bars). And slice those per job. My suggestion is that we do the actual visualization with matplotlib because it's super easy to output that from pandas data sets.<br>
</blockquote><div><br></div><div>The one thing that the current graph does, that weekly and daily failure rates don't show, is a sudden spike in one of the lines. If you stare at the current graphs for long enough and can read through the noise, you can see when the gate collectively crashes or if just the neutron related gates start failing. So I think one more graph is needed.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Basically we'll be mining Elastic Search -> Pandas TimeSeries -> transforms and analysis -> output tables and graphs. This is different enough from our current jquery graphing that I want to get ACKs before doing a bunch of work here and finding out people don't like it in reviews.<br>
<br>
Also in this process upgrade the metadata that we provide for each of those bugs so it's a little more clear what you are looking at.<br></blockquote><div><br></div><div>For example?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
= Take over of /recheck =<br>
<br>
There is still a bunch of useful data coming in on "recheck bug ####" data which hasn't been curated into ER queries. I think the right thing to do is treat these as a work queue of bugs we should be building patterns out of (or completely invalidating). I've got a preliminary gerrit bulk query piece of code that does this, which would remove the need of the daemon the way that's currently happening. The gerrit queries are a little long right now, but I think if we are only doing this on hourly cron, the additional load will be negligible.<br>
<br>
This would get us into a single view, which I think would be more informative than the one we currently have.<br></blockquote><div><br></div><div>treating /recheck as a work queue sounds great, but this needs a bit more fleshing out I think.</div>
<div><br></div><div>I imagine the workflow as something like this:</div><div><br></div><div>* State 1: Path author files bug saying 'gate broke, I didn't do it and don't know why it broke'.</div><div>* State 2: Someone investigates the bug and determines if bug is valid and if its a duplicate or not. root cause still isn't known.</div>
<div>* State 3: Someone writes a fingerprint for this bug and commits it to elastic-recheck.</div><div><br></div><div>Assuming we agree on this general workflow, it would be nice if /recheck distinguished between bugs in states 1 and 2, and there is no need to list bugs in state 3 as e-r bot will automatically tell a developer when he hits it.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
= Categorize all the jobs =<br>
<br>
We need a bit of refactoring to let us comment on all the jobs (not just tempest ones). Basically we assumed pep8 and docs don't fail in the gate at the beginning. Turns out they do, and are good indicators of infra / external factor bugs. They are a part of the story so we should put them in.<br>
</blockquote><div><br></div><div>Don't forget grenade</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
= Multi Line Fingerprints =<br>
<br>
We've definitely found bugs where we never had a really satisfying single line match, but we had some great matches if we could do multi line.<br>
<br>
We could do that in ER, however it will mean giving up logstash as our UI, because those queries can't be done in logstash. So in order to do this we'll really need to implement some tools - cli minimum, which will let us easily test a bug. A custom web UI might be in order as well, though that's going to be it's own chunk of work, that we'll need more volunteers for.<br>
<br>
This would put us in a place where we should have all the infrastructure to track 90% of the race conditions, and talk about them in certainty as 1%, 5%, 0.1% bugs.</blockquote><div><br></div><div><br></div><div>Horrah. multi line matches are two separate elasticSearch queries, where you match build_uuids. So to get the set of all hits of a multi line fingerprint you find the intersection between line_1 and line_2 where the key is build_uuid</div>
<div><div><br></div><div><table border="1" class="" style="border:0px solid rgb(221,204,238);border-collapse:collapse;color:rgb(0,0,0);font-family:sans-serif;font-size:16.363636016845703px"><tbody valign="top"><tr></tr></tbody></table>
</div><br><div></div></div><div><div><br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class=""><font color="#888888"><br>
<br>
-Sean<br>
<br>
-- <br>
Sean Dague<br>
Samsung Research America<br>
<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a> / <a href="mailto:sean.dague@samsung.com" target="_blank">sean.dague@samsung.com</a><br>
<a href="http://dague.net" target="_blank">http://dague.net</a><br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</font></span></blockquote></div><br></div></div>