Open Stack

On 09/12/13 14:03, Clint Byrum wrote:
> Excerpts from Zane Bitter's message of 2013-12-09 09:52:25 -0800:
>> On 09/12/13 06:31, Steven Hardy wrote:
>>> Hi all,
>>>
>>> So I've been getting concerned about $subject recently, and based on some
>>> recent discussions so have some other heat-core folks, so I wanted to start
>>> a discussion where we can agree and communicate our expectations related to
>>> nomination for heat-core membership (becuase we do need more core
>>> reviewers):
>>>
>>> The issues I have are:
>>> - Russell's stats (while very useful) are being used by some projects as
>>>     the principal metric related to -core membership (ref TripleO's monthly
>>>     cull/name&shame, which I am opposed to btw).  This is in some cases
>>>     encouraging some stats-seeking in our review process, IMO.
>>>
>>> - Review quality can't be measured mechanically - we have some folks who
>>>     contribute fewer, but very high quality reviews, and are also very active
>>>     contributors (so knowledge of the codebase is not stale).  I'd like to
>>>     see these people do more reviews, but removing people from core just
>>>     because they drop below some arbitrary threshold makes no sense to me.
>>
>> +1
>>
>> Fun fact: due to the quirks of how Gerrit produces the JSON data dump,
>> it's not actually possible for the reviewstats tools to count +0
>> reviews. So, for example, one can juice one's review stats by actively
>> obstructing someone else's work (voting -1) when a friendly comment
>> would have sufficed. This is one of many ways in which metrics offer
>> perverse incentives.
>>
>> Statistics can be useful. They can be particularly useful *in the
>> aggregate*. But as soon as you add a closed feedback loop you're no
>> longer measuring what you originally thought - mostly you're just
>> measuring the gain of the feedback loop.
>>
>
> I think I understand the psychology of stats and incentives, and I know
> that this _may_ happen.
>
> However, can we please be more careful about how this is referenced?
> Your message above is suggesting the absolute _worst_ behavior from our
> community. That is not what I expect, and I think anybody who was doing
> that would be dealt with _swiftly_.

Sorry for the confusion, I wasn't trying to suggest that at all. FWIW I 
haven't noticed anyone gaming the stats (maybe I haven't been looking at 
enough reviews ;). What I have noticed is that every time I leave a +0 
comment on a patch, I catch myself thinking "this won't look good on the 
stats" - and then I continue on regardless. If somebody who wasn't core 
but wanted to be were to -1 the patch instead in similar circumstances, 
then I wouldn't blame them in the least for responding to that incentive.

My point, and I think Steve's, is that we should be careful how we *use* 
the stats, so that folks won't feel this pressure. It's not at all about 
calling anybody out, but I apologise for not making that clearer.

cheers,
Zane.