Open Stack

On Fri, Sep 19, 2014 at 09:13:29AM -0400, Sean Dague wrote:
> I've spent the better part of the last 2 weeks in the Nova bug tracker
> to try to turn it into something that doesn't cause people to run away
> screaming. I don't remember exactly where we started at open bug count 2
> weeks ago (it was north of 1400, with > 200 bugs in new, but it might
> have been north of 1600), but as of this email we're at < 1000 open bugs
> (I'm counting Fix Committed as closed, even though LP does not), and ~0
> new bugs (depending on the time of the day).
> 
> == Philosophy in Triaging ==
> 
> I'm going to lay out the philosophy of triaging I've had, because this
> may also set the tone going forward.
> 
> A bug tracker is a tool to help us make a better release. It does not
> exist for it's own good, it exists to help. Which means when evaluating
> what stays in and what leaves we need to evaluate if any particular
> artifact will help us make a better release. But also more importantly
> realize that there is a cost for carrying every artifact in the tracker.
> Resolving duplicates gets non linearly harder as the number of artifacts
> go up. Triaging gets non-linearly hard as the number of artifacts go up.
> 
> With this I was being somewhat pragmatic about closing bugs. An old bug
> that is just a stacktrace is typically not useful. An old bug that is a
> vague sentence that we should refactor a particular module (with no
> specifics on the details) is not useful. A bug reported against a very
> old version of OpenStack where the code has changed a lot in the
> relevant area, and there aren't responses from the author, is not
> useful. Not useful bugs just add debt, and we should get rid of them.
> That makes the chance of pulling a random bug off the tracker something
> that you could actually look at fixing, instead of mostly just stalling out.
> 
> So I closed a lot of stuff as Invalid / Opinion that fell into those camps.
> 
> == Keeping New Bugs at close to 0 ==
> 
> After driving the bugs in the New state down to zero last week, I found
> it's actually pretty easy to keep it at 0.
> 
> We get 10 - 20 new bugs a day in Nova (during a weekday). Of those ~20%
> aren't actually a bug, and can be closed immediately. ~30% look like a
> bug, but don't have anywhere near enough information in them, and
> flipping them to incomplete with questions quickly means we have a real
> chance of getting the right info. ~10% are fixable in < 30 minutes worth
> of work. And the rest are real bugs, that seem to have enough to dive
> into it, and can be triaged into Confirmed, set a priority, and add the
> appropriate tags for the area.
> 
> But, more importantly, this means we can filter bug quality on the way
> in. And we can also encourage bug reporters that are giving us good
> stuff, or even easy stuff, as we respond quickly.
> 
> Recommendation #1: we adopt a 0 new bugs policy to keep this from
> getting away from us in the future.
> 
> == Our worse bug reporters are often core reviewers ==
> 
> I'm going to pick on Dan Prince here, mostly because I have a recent
> concrete example, however in triaging the bug queue much of the core
> team is to blame (including myself).
> 
> https://bugs.launchpad.net/nova/+bug/1368773 is a terrible bug. Also, it
> was set incomplete and no response. I'm almost 100% sure it's a dupe of
> the multiprocess bug we've been tracking down but it's so terse that you
> can't get to the bottom of it.
> 
> There were a ton of 2012 nova bugs that were basically "post it notes".
> Oh, "we should refactor this function". Full stop. While those are fine
> for personal tracking, their value goes to zero probably 3 months after
> they are files, especially if the reporter stops working on the issue at
> hand. Nova has plenty of "wouldn't it be great if we... " ideas. I'm not
> convinced using bugs for those is useful unless we go and close them out
> aggressively if they stall.
> 
> Also, if Nova core can't file a good bug, it's hard to set the example
> for others in our community.
> 
> Recommendation #2: hey, Nova core, lets be better about filing the kinds
> of bugs we want to see! mkay!

Not just Nova core. From my observation, most often bugs are often filed
in a hurry while going through a Elastic Search/logstash, etc.

I updated the below wiki that I wrote a while ago with a reference to
this email (thanks for taking the time to write it):

    https://wiki.openstack.org/wiki/BugFilingRecommendations

Reproducing the content of *why* the time to file a more informative bug
(for those who prefer to read plain-text). Nothing radically new, just a
gentle reminder:

  - Useful for new test engineers who do not have all the context of a
    bug.
  - Useful for documentation writers to help them write correct errata
    text/release notes.
  - Useful for non-technical folks reading the bugs/RFEs. Clear
    information saves a heck of a lot of time!
  - Useful for folks like product and program managers who’re always not
    in the trenches.
  - Useful for downstream support organizations.
  - Should there be a regression years later, having all the info to
    test/reproduce in the bug, right there makes your day.
  - Reduces needless round-trips of NEEDINFO between various parties.
  - Useful for new users referring to these bugs in a different context.

Overall, a fine bug reading experience!

Of course, not all bugs deserve the same attention, but just that we
should refrain the temptation to file utterly useless bugs with just a
stacktrace and no other useful context whatsoever - which puts all the
burden on the traiger to chase down.

--
/kashyap