Open Stack

Hi,

The really short version of this is:

 * Start using bug numbers after rechecks or reverifies, like:
     "recheck bug #####"
     "reverify bug #####"
 * Use http://status.openstack.org/rechecks/ to find likely bugs

Here's why:

We run a lot of jobs in Jenkins.  Between 1000 and 4000 per day.  That
presents a lot of opportunities for non-deterministic code (in programs
or in tests) to show itself.  And the fact that we gate changes on many
of those jobs means we notice it when that happens.

Sometimes a nondeterministic bug is rare, such that it doesn't really
bother anyone and it won't be seen on a subsequent recheck.  In that
case, we've lost an opportunity to collect data on a real-world bug.
Other times it happens so often that it affects nearly everyone in the
project, and we need to mobilize to find and fix the bug ASAP.

This change is designed to help collect information that can be used to
help identify bugs that are affecting people the most, and communicate
that as quickly as possible to everyone affected.

Every time a recheck/reverify comment is left with a bug number
attached, this page:

  http://status.openstack.org/rechecks/

will update.  It includes information such as when the bug was first and
most recently seen, how many times it has prompted someone to recheck a
change, what projects it has affected, and what changes in Gerrit it has
affected.

That page can be used by people working to fix bugs to help prioritize
work.  It is sorted by frecency[1], meaning that new bugs or high-impact
bugs bubble up to the top, so if your change fails a test, you can look
at that page and easily see if it already lists a likely cause.

If you don't see a bug on that page, please go to Launchpad to search
for an existing bug to use for the recheck, and if you don't find one,
please file a new bug.  Be sure to include some key output from the logs
so that others who see the same bug will be able to identify it.  Try to
file the bug against the right project, but also remember that it's easy
to add multiple projects in Launchpad and add or remove them later as
needed.

If the problem relates to infrastructure, such as a Jenkins exception,
inability to download something from the network, etc, please file it
against the openstack-ci project.

There are times when it makes sense to recheck a change without a bug:
for instance, if you know the branch or some dependency has changed
since the last check, or you have just made a draft change visible.  In
those cases, you can leave a comment with the syntax "recheck no bug" or
"reverify no bug".  Please only do this if there really is no bug --
that's what you're asserting to other developers when you leave such a
comment.

The new syntax is currently recognized by zuul, and the recheck status
page is being updated.  Starting soon, the comments from Jenkins that
are left in Gerrit will include a link to this wiki page:

  http://wiki.openstack.org/GerritJenkinsGithub#Test_Failures

which contains a summary of this information and a reference for the new
syntax, so it will be easily accessible in the future if you need to
reference it.  Starting in a week, the new syntax will be required.

I hope the new system proves useful, and thanks for your help!

[1] http://en.wiktionary.org/wiki/frecency

-Jim