Open Stack

First of all, thank you Emilien for bringing this up, and thank you Matt for
confirming our commitment to collaborate with puppet-openstack and other
Puppet modules that Fuel developers consider upstream.

I'd like to add some more concrete examples of what Fuel team has
already done, is doing, and what more we can do to improve our
collaboration with other Puppet developers.

As Matt has pointed out, sharing bugfixes for forked Puppet modules is
already a requirement for all Fuel developers:

https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Fuel_library_for_puppet_manifests

Here are the key bits that are specifically designed to simplify
reconciling of changes between Fuel and its upstream:

1) "If you are adding a module that is the work of another project and
is already tracked in separate repo (...) review should also contain the
commit hash from the upstream repo in the commit message". Using this
reference to upstream commit, you can cleanly and automatically isolate
Fuel specific changes to that module into a patch series with two useful properties:

   a) You can associate each deviant line with a commit in fuel-library
   repository that would refer to specific LP bug or blueprint and
   otherwise explain the change.

   b) The whole patch series can be cleanly applied on top of the
   specified commit in upstream git. This means you can automatically
   graft a branch made out of the patch series onto upstream git, and
   then use git rebase to make that branch mergeable with the current
   upstream git head.

2) "submit patch to upstream module first, and once merged or +2
recieved from a core reviewer, the patch should be backported to Fuel
library as well". Aside from the obvious benefit of immediate
contribution to upstream, this rule helps to keep Fuel specific changes
confined within Fuel-specific modules, and discourages arbitrary
deviations of forked modules from upstream.

It's important to understand that Fuel's "How to contribute" guide is
not just a set of recommendations for external contributors: it is the
primary definition of the engineering process that all Fuel developers,
within and without Mirantis, are expected to follow. If you come across
a violation of these rules in Fuel, it is a bug, you're more than
welcome to report it Launchpad, and Fuel developers will address it.

I understand that even with these rules in place the process isn't
perfect, even if it were followed flawlessly. If you have ideas on what
else we can tweak to make upstream's life easier, please share, we're
always looking for ways to improve our processes.

I hope you understand that there are some tradeoffs that are not worth
considering. While we can (and do) set aside some time in every release
to reconcile our local changes with upstream, we can't allow any
external factors to affect our ability to meet our own deadlines. For
example, we can and should push all our local changes to upstream, but
we can't hold back a fix for a Critical bug in Fuel until it is found
acceptable by upstream. We can further minimize the delta between our
fork and an upstream module, even eliminate the delta altogether when
possible, but we have to keep a copy of the module in a repository that
we control. We can and should make a more efficient use of the time our
engineers spend interacting with upstream, but only as long as that does
not subtract from the effort we put into Fuel project's priorities.

At the same time, it's important for Fuel developers to understand that
even though Fuel source code is publicly available and tracking and
minimizing deviation from upstream is encoded in our engineering
process, it doesn't mean there's no room for improvement. Saying "don't
worry, be happy, we've got it under control, the ball is on your side"
is not good enough.

P.S. There's been more emails on the thread while I was writing this,
and some points raised are not covered in this email, I'll try to
address them in another followup.

-- 
Dmitry Borodaenko



On Thu, Jun 11, 2015 at 05:36:57PM +0300, Matthew Mosesohn wrote:
> Hi Emilien,
> 
> I can see why you might be unhappy with Fuel's actions with regards to
> the OpenStack Puppet modules. You could make this argument about many
> components in Fuel. The heart of the matter is that we bundle the
> upstream OpenStack Puppet modules with all the other modules,
> developed both upstream and by Fuel's developers in one single git
> repository. This decision, along with all the other decisions to put
> Fuel's components under its own repositories, was intended to add
> stability and granular control to the product. I'm not saying it's the
> most community-oriented approach, but Fuel would have never evolved
> and matured without it. The attribution in commits is lost because our
> directory namespace differs such that it can't just be merged cleanly.
> Revisiting submodules is an option, but it led to maintenance problems
> in the past.
> 
> Secondly, I'd like to point out that Fuel is not so different from
> what other teams are doing. At the Summit, I heard from others who all
> maintain internal Gerrits and internal forks of the modules. The
> difference is that Fuel is being worked on in the open in StackForge.
> Anyone is free to contribute to Fuel as he or she wishes, take our
> patches, or review changesets.
> 
> Starting in October 2014, the Fuel team has adopted a policy that we
> cannot merge any patches into the core Puppet OpenStack modules of
> Fuel without submitting a patch or at least a bug upstream. Our
> reviewers block patches consistently. The truth is that the upstream
> modules are quite excellent and we don't need to make changes so
> often. Our goal is to work with upstream modules or in the issue where
> upstream integration is impossible, we place that config in our own
> separate modules.
> 
> The point you raised about fixing bugs in Fuel and not in Puppet
> OpenStack is definitely valid and something we need to collaborate on.
> The first and easiest option when a bug is applicable to both, we
> could use Launchpad to assign bugs to both Fuel project and
> puppet-$project so it gains visibility. If a bug is discovered in
> Puppet OpenStack after it's been reported and/or fixed in Fuel, then
> it would be best to ping someone in #fuel-dev on IRC and we can try to
> figure out how to get this applied upstream correctly. Our best
> results come from fixing upstream and I want to make sure that is
> clear.
> 
> If you have specific bugs or commits you'd like to discuss, let's work
> them out. I believe I can get the Fuel Library team to agree to do a
> walk through all the open bugs in Puppet OpenStack and see if we have
> any related fixes or bug reports.
> 
(...)
> > On Thu, Jun 11, 2015 at 9:12 AM, Emilien Macchi <emilien at redhat.com> wrote:
> >> Hi,
> >>
> >> Before reading this e-mail, please keep in mind:
> >>
> >> * I have a lot of admiration for Fuel and since I'm working on OpenStack
> >> Installers (at eNovance and now Red Hat), Fuel is something I always
> >> consider a good product.
> >> * This e-mail is about Fuel and Puppet, nothing about Mirantis.
> >> * I'm writing on behalf of my thoughts, and not on our group.
> >> * I'm using open mailing-list for open discussion. There is not bad
> >> spirit in this e-mail and I want to have a productive thread.
> >>
> >> I have some concerns I would like to share with you and hopefully find
> >> some solutions together.
> >>
> >> Since I've been working on Puppet OpenStack (2 years now), I see some
> >> situations that happen - according to me - too often:
> >>
> >> * A bug is reported in both Fuel Library and the Puppet module having
> >> trouble. A patch is provided in Fuel Library (your fork of Puppet
> >> OpenStack modules) but not in Puppet upstream module. That means you fix
> >> the bug for Fuel, and not for Puppet OpenStack community. It does not
> >> happen all the time but quite often.
> >>
> >> * A patch is submitted in a Puppet module and quite often does not land
> >> because there is no activity, no tests or is abandonned later because
> >> fixed in Fuel Library. I've noticed the patch is fixed in Fuel Library
> >> though.
> >>
> >> * RAW copy/paste between upstream modules code and your forks. In term
> >> of Licensing, I'm even not sure you have the right to do that (I'm not a
> >> CLA expert though) but well... in term of authorship and statistics on
> >> code, I'm not sure it's fair. Using submodules with custom patches would
> >> have been great to respect the authors who created the original code and
> >> you could have personalize the manifests.
> >>
> >> Note: you can see that I don't give any example because I'm not here to
> >> blame people or judge anyone.
> >>
> >> So the goal of my e-mail is to open the discussion and have a *real*
> >> collaboration between Fuel team which seems to have a lot of good Puppet
> >> engineers and Puppet OpenStack team.
> >>
> >> We had this kind of discussion at the Summit (in Vancouver and also
> >> Paris, and even before). Now I would like to officialy know if you are
> >> interested or not to be more involved.
> >> I'm also open at any feedback about Puppet OpenStack group and if
> >> something blocks you to contribute more.
> >>
> >> We have the same goals, having Puppet modules better. I think it can be
> >> win/win: you have less diff with upstream and we have more hands in our
> >> module maintenance.
> >> Thank you for reading so far, and I'm looking forward to reading from you.