Open Stack

It has been 5 weeks since Emilien has asked Fuel developers to contribute more
actively to Puppet OpenStack project [0]. We had a lively discussion on
openstack-dev, myself and other Fuel developers proposed some concrete steps
that could be done to reconcile the two projects, the whole thread was reported
and discussed on Linux Weekly News [1], and the things were looking up.

[0] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066544.html
[1] https://lwn.net/Articles/648331/

And now, 5 weeks later, Emilien has reported to the Technical Committee that
there has been no progress on reconciliation between Fuel and Puppet OpenStack,
and used his authority as a PTL to request that the Fuel's proposal to join
OpenStack [2] is rejected.

[2] https://review.openstack.org/199232

In further comments on the same review, Emilien has claimed that there's
"clearly less" contribution to Puppet OpenStack from Fuel developers than
before, and even brought up an example of a review in puppet-horizon that was
proposed and then abandoned by Fuel team [3]. Jay went as far as calling that
example "an obvious failure of working with the upstream Puppet-OpenStack
community".

[3] https://review.openstack.org/198119

Needless to say, I found all these claims deeply disturbing, and had to look
closely into what's going on.

The case of the puppet-horizon commit has turned out to be surprisingly
obvious.

Even before looking into the review comments, I could see a technical reason
for abandoning the commit: if there is a bug in a component, fixing that bug in
the package is preferrable to fixing it in puppet, because it allows anybody to
benefit from the fix, not just the people deploying that package with puppet.

And if you do look at the review in question, you'll find that immediately (14
minutes, and that at 6pm on Friday night!) after Jay has asked in the comments
to the review why it was abandoned, the commit author from the Fuel team has
explained that this patch was a workaround for a packaging problem, and that
this was pointed out in the review by a Horizon core reviewer more than a week
ago, and later corroborated by a Puppet OpenStack core reviewer. Further
confirming that fixing this in the package instead of in puppet-horizon was an
example of Fuel developers agreeing with other Puppet OpenStack contributors
and doing the right thing.

Emilien has clearly found this case important enough to report to the TC, and
yet didn't find it important enough to simply ask Fuel developers why they
chose to abandon the commit. I guess you can call that an obvious failure to
work together.

Looking further into Fuel team's reviews for puppet-horizon, I found another
equally disturbing example [4].

[4] https://review.openstack.org/190548

Here's what I see in this review:

a) Fuel team has spent more than a month (since June 11) on trying to land this
patch.

b) 2 out of 5 negative reviews are from Fuel team, proving that Fuel developers
are not "rubberstamping" each other's commits as was implied by Emilien's
comments on the TC review above.

c) There was one patch set that was pushed 3 business days after a negative
review, all other patch sets (11 total) were pushed no later than next day
after a negative review.

All in all, I see great commitment and effort on behalf of Fuel team,
eventually awarded by a +2 from Michael Chapman.

On the same day (June 30), Emilien votes -2 for a correction in a comment, and
walks away from the review for good. 18 days and 4 patch sets and 2 outstanding
+1's later, the review remains blocked by that -2. Reaching out on
#puppet-openstack [5] didn't help, either.

[5] http://irclog.perlgeek.de/puppet-openstack/2015-07-08#i_10867124

At the same time, Emilien has commented on the TC review that the only metric
he considers reflective of Fuel's contribution to Puppet OpenStack is merged
commits. Isn't that a Catch-22 situation, requesting more merged commits and
refusing to merge them?

When I compare the trends in the number of patch sets pushed by Fuel developers
[6] against the number of merged commits [7], I see the same picture. It's
really obvious when comparing both graphs side by side.

[6] http://stackalytics.com/?module=puppetopenstack-group&metric=patches&company=mirantis
[7] http://stackalytics.com/?module=puppetopenstack-group&metric=commits&company=mirantis

Between May and July, the peak number of patchsets in Puppet OpenStack from all
contributors increased by a factor of 1.78, while patchsets associated with
Mirantis jumped up by a factor of 6.75. At the same time, merged commits from
all contributors increased by 2.44 (meaning 30% less patch sets per commit than
in May), for Mirantis the factor is only 2.0 (meaning 3 times as many patch
sets per commit as in May).

Just look at the attached picture to see how obviously massive is the increase
in the effort that Fuel team contributes to Puppet OpenStack. One can argue for
different ways to explain why this effort does not convert into merged commits,
but based on two examples above I find it very hard to believe that it can be
explained by a lack of trying.

In fact, this is exactly the situation that I have predicted in the original
thread when I said that commmitment to collaborate has to be mutual [8].

[8] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066731.html

Fuel team can commit to reporting bugs and posting patches, Fuel team can spend
reasonable amount of time and effort to respond to review comments, but all
that is worth nothing if there's no commitment from Puppet OpenStack team to
review and merge these patches in a timely manner.

Looking through the collaboration thread, the only commitment I see is
Emilien's promise that "if you submit a good patch now, it will land in maximum
one week or so", and his comment on the TC review: "The end of the thread was
kind of consensus to me and I was pretty happy to see Dmitry was really open to
the discussion".

Clearly this kind of consensus is not working out, and I think at least one
reason is that my proposal for Puppet OpenStack developers to take over stalled
commits was rejected. One way or another, cases like "16 days since the first
patch set that had no negative reviews" that obviously violate the "land within
a week" commitment have to be identified and prioritized. Being on the hook to
take them over if they don't land on time is one way to introduce a motivation
to make that happen. There could be other ways to address it, for example
having a review dashboard that highlights stuck reviews (as I proposed in the
same email) would help add some visibility into the problem.

Please don't get me wrong. In the same reviews I've highlighted above I see
excellent example of collaboration from Puppet OpenStack team, insightful and
timely comments, and I'm not denying the effort the team spends on onboarding
new contributors flooding in from Fuel. But I don't see how this situation can
be called "no progress" and "reduced contribution from Fuel" when making
further progress is limited by Puppet OpenStack team's ability to review
incoming patches much more than by Fuel team's commitment to proposing and
updating them.

-- 
Dmitry Borodaenko
-------------- next part --------------
A non-text attachment was scrubbed...
Name: puppetopenstack-mirantis-patches.png
Type: image/png
Size: 20073 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150718/c7c7a7c5/attachment.png>