Open Stack

I'm currently in holidays but I could not resist to take some time and
reply.

On 07/18/2015 06:32 PM, Dmitry Borodaenko wrote:
> It has been 5 weeks since Emilien has asked Fuel developers to
> contribute more
> actively to Puppet OpenStack project [0]. We had a lively discussion on
> openstack-dev, myself and other Fuel developers proposed some concrete
> steps
> that could be done to reconcile the two projects, the whole thread was
> reported
> and discussed on Linux Weekly News [1], and the things were looking up.
> 
> [0]
> http://lists.openstack.org/pipermail/openstack-dev/2015-June/066544.html
> [1] https://lwn.net/Articles/648331/
> 
> And now, 5 weeks later, Emilien has reported to the Technical Committee
> that
> there has been no progress on reconciliation between Fuel and Puppet
> OpenStack,
> and used his authority as a PTL to request that the Fuel's proposal to join
> OpenStack [2] is rejected.

1) What you're writing here is not correct.
Please read again my first comment on the Governance patch:

"They are making progress and I'm sure they are doing their best."

I've been following all your actions since our *thread*, and so far all
of this seems good to me, I'm honestly satisfied to know that you guys
have plans.

2) The word 'authority' is also wrong. I'm not part of TC and my vote
does not count at all for the final decision. If Fuel has to be an
OpenStack project, TC is free to vote by themselves without me and it
seems they already negatively voted before my thoughts.

3) I -1 the governance patch because to me, it seems too early for
having Fuel part of OpenStack. We ran that discussion 5 weeks ago, a lot
of actions have been taken for I hope TC will wait for actual results
(mainly highlighted in the review itself).
Let me quote my first comment on the patch that clearly show I'm not
against the idea:
"I sincerely hope they'll realize what they plan to do, which is being
part of our community like other projects already succeed to do."


> [2] https://review.openstack.org/199232
> 
> In further comments on the same review, Emilien has claimed that there's
> "clearly less" contribution to Puppet OpenStack from Fuel developers than
> before, and even brought up an example of a review in puppet-horizon
> that was
> proposed and then abandoned by Fuel team [3]. Jay went as far as calling
> that
> example "an obvious failure of working with the upstream Puppet-OpenStack
> community".

Andrew took the best metric in your advantage... the review metric,
which is something given to anyone having signed the CLA.

I would rather focus on patchset, commits, bug fix, IRC, ML etc... which
really show collaboration in a group.
And I honestly think it's making progress, even though the numbers.

> [3] https://review.openstack.org/198119
> 
> Needless to say, I found all these claims deeply disturbing, and had to
> look
> closely into what's going on.
> 
> The case of the puppet-horizon commit has turned out to be surprisingly
> obvious.
> 
> Even before looking into the review comments, I could see a technical
> reason
> for abandoning the commit: if there is a bug in a component, fixing that
> bug in
> the package is preferrable to fixing it in puppet, because it allows
> anybody to
> benefit from the fix, not just the people deploying that package with
> puppet.

You are not providing official Ubuntu packaging, but your own packages
mainly used by Fuel, while Puppet OpenStack modules are widely used by
OpenStack community.
Fixing that bug in Fuel packaging is the shortest & easiest way for you
to fix that, while we are really doing something wrong in puppet-horizon
about the 'compress' option.
So Fuel is now fixed and puppet-horizon broken.

> And if you do look at the review in question, you'll find that
> immediately (14
> minutes, and that at 6pm on Friday night!) after Jay has asked in the
> comments
> to the review why it was abandoned, the commit author from the Fuel team
> has
> explained that this patch was a workaround for a packaging problem, and
> that
> this was pointed out in the review by a Horizon core reviewer more than
> a week
> ago, and later corroborated by a Puppet OpenStack core reviewer. Further
> confirming that fixing this in the package instead of in puppet-horizon
> was an
> example of Fuel developers agreeing with other Puppet OpenStack
> contributors
> and doing the right thing.

This kind of bug has to be fixed between Horizon & Puppet team, and not
with some "workaround" in packaging tool.
We don't really want "workarounds" I guess, do we?

> Emilien has clearly found this case important enough to report to the
> TC, and
> yet didn't find it important enough to simply ask Fuel developers why they
> chose to abandon the commit. I guess you can call that an obvious
> failure to
> work together.
> 
> Looking further into Fuel team's reviews for puppet-horizon, I found
> another
> equally disturbing example [4].
> 
> [4] https://review.openstack.org/190548
> 
> Here's what I see in this review:
> 
> a) Fuel team has spent more than a month (since June 11) on trying to
> land this
> patch.
> 
> b) 2 out of 5 negative reviews are from Fuel team, proving that Fuel
> developers
> are not "rubberstamping" each other's commits as was implied by Emilien's
> comments on the TC review above.
> 
> c) There was one patch set that was pushed 3 business days after a negative
> review, all other patch sets (11 total) were pushed no later than next day
> after a negative review.
> 
> All in all, I see great commitment and effort on behalf of Fuel team,
> eventually awarded by a +2 from Michael Chapman.
> 
> On the same day (June 30), Emilien votes -2 for a correction in a
> comment, and
> walks away from the review for good. 18 days and 4 patch sets and 2
> outstanding
> +1's later, the review remains blocked by that -2. Reaching out on
> #puppet-openstack [5] didn't help, either.
> 
> [5] http://irclog.perlgeek.de/puppet-openstack/2015-07-08#i_10867124

I -2 for this reason: this patch already had a +2 so it was closed to be
merged by someone else, while the patch was *breaking backward
compatiblity* in puppet-horizon, because Vasyl was changing the default
cache driver to force users to use Memcached by default.
We *never* break backward compatibility (this is a general rule in
OpenStack) and *never* change OpenStack default configuration (general
rule in Configuration management), but rather provide the interface
(called Puppet parameters) to change it.

> At the same time, Emilien has commented on the TC review that the only
> metric
> he considers reflective of Fuel's contribution to Puppet OpenStack is
> merged
> commits. Isn't that a Catch-22 situation, requesting more merged commits
> and
> refusing to merge them?
> 
> When I compare the trends in the number of patch sets pushed by Fuel
> developers
> [6] against the number of merged commits [7], I see the same picture. It's
> really obvious when comparing both graphs side by side.
> 
> [6]
> http://stackalytics.com/?module=puppetopenstack-group&metric=patches&company=mirantis
> 
> [7]
> http://stackalytics.com/?module=puppetopenstack-group&metric=commits&company=mirantis
> 
> 
> Between May and July, the peak number of patchsets in Puppet OpenStack
> from all
> contributors increased by a factor of 1.78, while patchsets associated with
> Mirantis jumped up by a factor of 6.75. At the same time, merged commits
> from
> all contributors increased by 2.44 (meaning 30% less patch sets per
> commit than
> in May), for Mirantis the factor is only 2.0 (meaning 3 times as many patch
> sets per commit as in May).
> 
> Just look at the attached picture to see how obviously massive is the
> increase
> in the effort that Fuel team contributes to Puppet OpenStack. One can
> argue for
> different ways to explain why this effort does not convert into merged
> commits,
> but based on two examples above I find it very hard to believe that it
> can be
> explained by a lack of trying.
> 
> In fact, this is exactly the situation that I have predicted in the
> original
> thread when I said that commmitment to collaborate has to be mutual [8].

/me happy
I'll see the review backlog with Puppet group and see if we lack in reviews.

> [8]
> http://lists.openstack.org/pipermail/openstack-dev/2015-June/066731.html
> 
> Fuel team can commit to reporting bugs and posting patches, Fuel team
> can spend
> reasonable amount of time and effort to respond to review comments, but all
> that is worth nothing if there's no commitment from Puppet OpenStack
> team to
> review and merge these patches in a timely manner.

Please consider our group is part of a few people comparing to Fuel's
team, and we are not all working 100% on Puppet OpenStack (I'm lucky, I
do). Please take that in consideration.

> Looking through the collaboration thread, the only commitment I see is
> Emilien's promise that "if you submit a good patch now, it will land in
> maximum
> one week or so", and his comment on the TC review: "The end of the
> thread was
> kind of consensus to me and I was pretty happy to see Dmitry was really
> open to
> the discussion".
> 
> Clearly this kind of consensus is not working out, and I think at least one
> reason is that my proposal for Puppet OpenStack developers to take over
> stalled
> commits was rejected. One way or another, cases like "16 days since the
> first
> patch set that had no negative reviews" that obviously violate the "land
> within
> a week" commitment have to be identified and prioritized. Being on the
> hook to
> take them over if they don't land on time is one way to introduce a
> motivation
> to make that happen. There could be other ways to address it, for example
> having a review dashboard that highlights stuck reviews (as I proposed
> in the
> same email) would help add some visibility into the problem.

We have weekly review/bug triage, feel free to participate, it happens
during our meetings.

> Please don't get me wrong. In the same reviews I've highlighted above I see
> excellent example of collaboration from Puppet OpenStack team,
> insightful and
> timely comments, and I'm not denying the effort the team spends on
> onboarding
> new contributors flooding in from Fuel. But I don't see how this
> situation can
> be called "no progress" and "reduced contribution from Fuel" when making
> further progress is limited by Puppet OpenStack team's ability to review
> incoming patches much more than by Fuel team's commitment to proposing and
> updating them.
> 

Please don't get me wrong, but you're aggravating the situation.
We are having much more collaboration than before.
Fuel's team is much more involved on reviews as you highlighted, and
also on IRC, which is good.
Of course there are still technical things that will be improved with
more experience in working with Puppet OpenStack modules, but so far all
Puppet grouped noticed more collaboration I think.

I totally assume my -1 for adding Fuel to OpenStack because it's too
early now, even all your recent efforts.

Now, I think we just need you to let us more time, so your engineers and
our group can work together. I'm pretty sure the things will go
naturally like they happened for our other contributors.

Rather than doing yet another very long thread, maybe can we arrange a
public meeting on IRC and we can do some triage together. After my
holidays, I can even organize a 1 or 2 hour meeting so we can solve our
first blockers and make progress on both sides. I'll try to get
attention from our group if some help is needed.
What do you think?

Best regards,

Post-scriptum: I'm back online on July 23.
---
Emilien Macchi