Open Stack

Excerpts from John Griffith's message of 2013-08-27 09:42:37 -0700:
> On Tue, Aug 27, 2013 at 10:26 AM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> 
> > I wonder if there's any sort of automation we can apply to this, for
> > example having known rechecks have "signatures" and if a failure matches
> > the signature it auto applies the recheck.
> >
> 
> I think we kinda already have that, the recheck list and the bug ID
> assigned to it no?  Automatically scanning said list and doing the recheck
> automatically seems like overkill in my opinion.  At some point human
> though/interaction is required and I don't think it's too much to ask a
> technical contributor to simply LOOK at the output from the test runs
> against their patches and help out a bit. At the very least if you didn't
> test your patch yourself and waited for Jenkins to tell you it's broken I
> would hope that a submitter would at least be motivated to fix their own
> issue that they introduced.
> 

It is worth thinking about though, because "ask a technical contributor
to simply LOOK" is a lot more expensive than "let a script confirm the
failure and tack it onto the list for rechecks".

Ubuntu has something like this going for all of their users and it is
pretty impressive.

Apport and/or whoopsie see crashes and look at the
backtraces/coredumps/etc and then (with user permission) submit a
signature to the backend. It is then analyzed and the result is this:

http://errors.ubuntu.com/

Known false positives are shipped along side packages so that they do
not produce noise, and known points of pain for debugging are eased by
including logs and other things in bug reports when users are running
the dev release. This results in a much better metric for what bugs to
address first. IIRC update-manager also checks in with a URL that is
informed partially by this data about whether or not to update packages,
so if there is a high fail rate early on, the server side will basically
signal update-manager "don't update right now".

I'd love to see our CI system enhanced to do all of the pattern
matching to group failures by common patterns, and then when a technical
contributor looks at these groups they have tons of data points to _fix_
the problem rather than just spending their precious time identifying it.

The point of the recheck system, IMHO, isn't to make running rechecks
easier, it is to find and fix bugs.