Open Stack

I'be seen very obvious examples of stats like this being gamed.  As an
example, overly trivial patches that add no value, shortly followed by
company press releases with how many developers are contributing to the
project.  I would also be careful with what stats we as a community put
emphasis on.  For example, I cringe every time someone lauds the number of
lines of code, as it is such a poor metric.

--
Chuck
On Apr 5, 2013 6:44 AM, "Mark McLoughlin" <markmc at redhat.com> wrote:

> On Fri, 2013-04-05 at 11:03 +0200, Thierry Carrez wrote:
> > Michael Still wrote:
> > > On Fri, Apr 5, 2013 at 2:47 AM, Stefano Maffulli <
> stefano at openstack.org> wrote:
> > >> On 04/04/2013 08:24 AM, Mark McLoughlin wrote:
> > >
> > > [snip]
> > >
> > >>> Frankly, though, I'm losing faith in this "number of commits"
> statistic
> > >>> being all that useful. Most of my commits were probably trivial cfg
> > >>> cleanups in Nova.
> > >>
> > >> Not one single number has meanings by itself but only acquires a
> meaning
> > >> inside a story. If you look at count of commits together with the
> number of
> > >> bugs closed over time, so you can see trends, for example, you may
> have an
> > >> idea of how much effort goes into stabilization vs new features.
> Looking at
> > >> commits and lines of code added/removed can also help understand
> where the
> > >> development effort is going. Or at least the numbers will seem to tell
> > >> something and force you to want to dive deeper.
> > >
> > > Yes. I am increasingly worried about employers managing to these
> > > numbers as well. I feel that the numbers that get discussed in the
> > > community are the ones employees are likely to optimise for.
> > >
> > > I'd like to see a more general discussion about what contributions we
> > > value as a community, and how we encourage those contributions.
> >
> > Yeah, I'm having the same concern, and the official "activity board" is
> > only making it more pressing:
> >
> > On one hand, it's extremely difficult to get those stats right. Tracking
> > where everyone works is painful (the Bitergia report, for example, still
> > counts me under "Rackspace"). The metrics themselves can be a bit
> > unfair: we already mentioned the issue with "number of commits"... but
> > others fail as well (like "bug activity": stable branch management will
> > result in opening a lot of bugs for process tracking, which will be an
> > order of magnitude bigger than the number of bugs opened due to QA
> > research).
> >
> > On the other hand, more and more people/companies want to be able to
> > brag about their OpenStack contribution, or want to not appear too bad
> > on those lists. That leads to the incentive to game the system, once
> > some stats are made "official".
> >
> > Until now, the stats posted were mostly one-shot and ad-hoc, to serve as
> > a reputational pressure encouraging people to contribute more,
> > especially in strategic areas. But if they become official and stable
> > metrics, they will be gamed, defeating their original purpose.
> >
> > Bruce Schneier puts it in better words than I do in "Liars and Outliers":
> >
> > "When you start measuring something and then judge people based on that
> > measurement, you encourage people to game the measurement instead of
> > doing whatever it is you wanted in the first place."
>
> Agree with everything you say. I don't think I've seen obvious examples
> of people trying to game the stats but, with the emphasis we're putting
> on these stats, it's only natural that people will start gaming them.
>
> Cheers,
> Mark.
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130405/266958d8/attachment.html>