Open Stack

Zane,

Backing the rules with incentives is very important, thanks for bringing
it up!

This is exactly why I'm trying to be conservative about how much
upstream integration overhead Fuel team can commit to: make it too
burdensome and it will fizzle out. On the other hand, unintrusive and
incremental process changes have a better chance of long-term adoption.

Earlier today, Bogdan has updated Fuel wiki with a recommendation to
create a Fuel plugin with a vanilla upstream module instead of adding a
forked version of the new module into Fuel core:

https://wiki.openstack.org/w/index.php?title=Fuel/How_to_contribute&diff=next&oldid=81244

This, together with the way you're describing handling upstream modules
in RDO, got me thinking: this can work (after all, it has no immediate
impact on the code that's already in fuel-library), and it will protect
us against adding more forked modules to the gap between Fuel and
upstream, but what can we do to start actually closing the gap instead
of simply stopping it from growing?

What I came up with is this. We currently have 3 levels of upstream
integration for different modules in fuel-library:

1) Fuel-specific modules that are useless outside of Fuel (e.g.
osnailyfacter).

2) Forked modules that have never been synced with upstream since their
initial inclusion (e.g. puppet-openstack).

3) Forked modules that have been synced with upstream using our new
policy (e.g. puppet-nova), but remain forked.

Restricting new modules to plugins introduces the next, fourth level of
integration: vanilla upstream modules that exactly match a specified
known-good tag or git commit id in the upstream repository. The problem
is, if you freeze it to a fixed version, only one side of the gap is
fixed: as upstream moves ahead and you stay behind the gap is still
growing.

To truly call it the next level of integration, we have to come up with
a way to keep up with upstream. And at the same time, we have to prevent
that from breaking Fuel: having to constantly deal with regressions
caused by upstream changes is exactly the kind of incentive that is
going to work against the whole idea of upstream integration.

And this brings us back to the reason why I said making Fuel CI vote on
upstream commits is a pre-requisite for using upstream modules directly.
If we can detect changes that would introduce regressions into Fuel
early (ideally before they are merged), tracking every upstream commit
becomes a reality. And when time lag behind upstream is eliminated
altogether, committing directly to upstream becomes less work than
maintaining a local fork.

The best part is, we don't have to wait for the Fuel pluggable
architecture to mature to the level where we can start moving Puppet
modules from core to plugins (currently this won't work for some key
modules such as keystone, mysql, etc).

All we need is to package fuel-library core into a set of binary
packages (rpm or deb), one per Puppet module, as Bogdan has proposed.
Start with a single git repo used as source package for all such binary
packages, with primary fuel-library package depending on module
packages.

Then, one at a time, extract a module into its own git-buildpackage [0]
repo and use gbp-pq [1] to convert Fuel specific patches into a quilt
patch queue (tell me if there are better ways to do that for RPMs).

[0] http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.html
[1] http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/man.gbp.pq.html

After that:

1. Fix fuel-library dependency to the first post-split revision of the
   module (which matches the pre-split code).

2. Set up CI to track upstream master branch and post non-voting Fuel CI
   test results to upstream gerrit.

3. Commit changes to Fuel core and upstream to get the Fuel test results
   to pass.

4. Turn on voting for Fuel CI tests in upstream.

5. Relax fuel-library dependency to unversioned, so that it takes the
   latest available revision of the module.

At that point, transition to full upstream integration is completed, for
that particular module.

What are we missing that's needed to make this work? Aside from
splitting fuel-library package, and making Fuel CI work on
openstack-infra?

-- 
Dmitry Borodaenko


On Fri, Jun 12, 2015 at 05:26:11PM -0400, Zane Bitter wrote:
> This thread kind of deteriorated a bit (though it looks like it's hopefully
> recovering), so I'd just like to add some observations.
> 
> What we have here is a classic case of a long-running fork, with all that
> that entails. In this case the fork is a public one, but that actually makes
> very little difference to the fundamentals. (I think it's great that
> Mirantis have chosen to develop Fuel in the open though! Kudos.)
> 
> The fact is that maintaining a fork is very expensive. And while it's
> expensive for the upstream community in terms of lost opportunities for bug
> fixes, it's far, far more expensive for the maintainers of the downstream
> fork. IMHO that's one of the reasons that permissive licenses like ASL2 have
> gained so much ground over the GPL - it didn't take very long for almost
> everyone to realise that there were more compelling reasons to contribute
> your code upstream than that you were compelled to by the license. I don't
> think a project like OpenStack could exist if they hadn't. It's simply
> better business, even if you consider the other upstream users to be
> competitors.
> 
> So I think both projects would benefit from more co-operation, but Fuel has
> by far the most to gain.
> 
> I see from the thread that a lot of well-intentioned policies have been put
> in place to try to improve co-operation, and it's actually not that
> surprising to see them not working that well because the incentives are
> wrong. When you set up a conflict between incentives and rules... incentives
> tend to win. (I probably don't need to try to prove this, because it was
> IMHO one of the great lessons of the communist experiment, and looking at
> the names in this thread I suspect that most of y'all at least know someone
> with direct experience of that.)
> 
> So at the moment committing a patch to Fuel is easy for a Fuel developer,
> whereas getting that same patch into upstream is hard. So it is much more
> likely that the downstream patch lands while the upstream patch languishes,
> despite the hidden cost that another Fuel developer will need to reconcile
> the two later. To get this to work, you need to make upstream the default
> (and therefore easiest) path to get changes included in Fuel.
> 
> Of course you will need a way to make urgent changes to your product without
> waiting for upstream. As an example, we do this in RDO Manager by
> maintaining patches on top of an upstream snapshot. (We do actually use Git
> - in a non-traditional way - as a tool to aid this process, but it's not
> really the point and there are many ways to tackle the problem.) The
> snapshot gets updated regularly, so changes that are committed upstream just
> show up without any extra work. If we need something urgently, we have to
> option to apply it as a patch, but our enthusiasm to do so is always
> tempered by the knowledge that if a change that is at least extremely
> similar does not land upstream then we are creating extra work for ourselves
> in the very near future. That's how we keep the incentives and policies
> aligned. (In this way, building a project around a library like this is very
> similar to building a downstream distribution around an upstream project. We
> use essentially the same techniques.)
> 
> And of course once the upstream becomes the default place to land patches,
> you'll very quickly stop thinking of upstream as 'them' and start thinking
> of them as 'us'. You'll start assimilating the ideas of what are and are not
> good coding standards so that you won't have to rework them nearly as much
> before they can be merged, and once you get involved in the community you'll
> have the opportunity to influence those ideas as well. Once everyone is up
> to speed I'm sure you'd see a lot of folks get added to core. Instead of
> upstream co-operation appearing to consume time that you don't have (which
> appears to be the problem at the moment), I'm quite sure that same people
> will be able to get a *lot* more done.
> 
> Tinkering with the current model by putting in place more policies or trying
> to offload work to the upstream openstack-puppet team will not work, and
> more importantly would not realise the same benefits to the Fuel team even
> if it did work.
> 
> The problem, of course, is that once you are on a long-running fork it takes
> a big up-front investment to get off it. (Ask anyone still running an
> OpenStack Folsom cloud ;) That can be hard to make a case for, especially
> when you have other priorities and the dividends take some time to appear. I
> think in this case it would be totally worth the investment, and I hope the
> Fuel team will consider making that investment.
> 
> As a bonus, it'll be more polite to the original authors of the code, it'll
> help everyone who is deploying OpenStack with Puppet (which is most people
> in the community), and it'll help Fuel users join a bigger critical mass of
> users so they can get better support from channels like ask.openstack.org.
> 
> cheers,
> Zane.
> 
> On 11/06/15 10:36, Matthew Mosesohn wrote:
> >Hi Emilien,
> >
> >I can see why you might be unhappy with Fuel's actions with regards to
> >the OpenStack Puppet modules. You could make this argument about many
> >components in Fuel. The heart of the matter is that we bundle the
> >upstream OpenStack Puppet modules with all the other modules,
> >developed both upstream and by Fuel's developers in one single git
> >repository. This decision, along with all the other decisions to put
> >Fuel's components under its own repositories, was intended to add
> >stability and granular control to the product. I'm not saying it's the
> >most community-oriented approach, but Fuel would have never evolved
> >and matured without it. The attribution in commits is lost because our
> >directory namespace differs such that it can't just be merged cleanly.
> >Revisiting submodules is an option, but it led to maintenance problems
> >in the past.
> >
> >Secondly, I'd like to point out that Fuel is not so different from
> >what other teams are doing. At the Summit, I heard from others who all
> >maintain internal Gerrits and internal forks of the modules. The
> >difference is that Fuel is being worked on in the open in StackForge.
> >Anyone is free to contribute to Fuel as he or she wishes, take our
> >patches, or review changesets.
> >
> >Starting in October 2014, the Fuel team has adopted a policy that we
> >cannot merge any patches into the core Puppet OpenStack modules of
> >Fuel without submitting a patch or at least a bug upstream. Our
> >reviewers block patches consistently. The truth is that the upstream
> >modules are quite excellent and we don't need to make changes so
> >often. Our goal is to work with upstream modules or in the issue where
> >upstream integration is impossible, we place that config in our own
> >separate modules.
> >
> >The point you raised about fixing bugs in Fuel and not in Puppet
> >OpenStack is definitely valid and something we need to collaborate on.
> >The first and easiest option when a bug is applicable to both, we
> >could use Launchpad to assign bugs to both Fuel project and
> >puppet-$project so it gains visibility. If a bug is discovered in
> >Puppet OpenStack after it's been reported and/or fixed in Fuel, then
> >it would be best to ping someone in #fuel-dev on IRC and we can try to
> >figure out how to get this applied upstream correctly. Our best
> >results come from fixing upstream and I want to make sure that is
> >clear.
> >
> >If you have specific bugs or commits you'd like to discuss, let's work
> >them out. I believe I can get the Fuel Library team to agree to do a
> >walk through all the open bugs in Puppet OpenStack and see if we have
> >any related fixes or bug reports.
> >
> >Best Regards,
> >Matthew Mosesohn
> >
> >On Thu, Jun 11, 2015 at 2:34 PM, Sanjay Upadhyay <saneax at gmail.com> wrote:
> >>+1 for the thread, I would also like to hear from Mirantis on this.
> >>
> >>The Fork on fuel/puppet has been actively seen patching and consolidation.It
> >>seems like parallel effort why not merge it.
> >>
> >>regards
> >>/sanjay
> >>
> >>On Thu, Jun 11, 2015 at 9:12 AM, Emilien Macchi <emilien at redhat.com> wrote:
> >>>
> >>>Hi,
> >>>
> >>>Before reading this e-mail, please keep in mind:
> >>>
> >>>* I have a lot of admiration for Fuel and since I'm working on OpenStack
> >>>Installers (at eNovance and now Red Hat), Fuel is something I always
> >>>consider a good product.
> >>>* This e-mail is about Fuel and Puppet, nothing about Mirantis.
> >>>* I'm writing on behalf of my thoughts, and not on our group.
> >>>* I'm using open mailing-list for open discussion. There is not bad
> >>>spirit in this e-mail and I want to have a productive thread.
> >>>
> >>>I have some concerns I would like to share with you and hopefully find
> >>>some solutions together.
> >>>
> >>>Since I've been working on Puppet OpenStack (2 years now), I see some
> >>>situations that happen - according to me - too often:
> >>>
> >>>* A bug is reported in both Fuel Library and the Puppet module having
> >>>trouble. A patch is provided in Fuel Library (your fork of Puppet
> >>>OpenStack modules) but not in Puppet upstream module. That means you fix
> >>>the bug for Fuel, and not for Puppet OpenStack community. It does not
> >>>happen all the time but quite often.
> >>>
> >>>* A patch is submitted in a Puppet module and quite often does not land
> >>>because there is no activity, no tests or is abandonned later because
> >>>fixed in Fuel Library. I've noticed the patch is fixed in Fuel Library
> >>>though.
> >>>
> >>>* RAW copy/paste between upstream modules code and your forks. In term
> >>>of Licensing, I'm even not sure you have the right to do that (I'm not a
> >>>CLA expert though) but well... in term of authorship and statistics on
> >>>code, I'm not sure it's fair. Using submodules with custom patches would
> >>>have been great to respect the authors who created the original code and
> >>>you could have personalize the manifests.
> >>>
> >>>Note: you can see that I don't give any example because I'm not here to
> >>>blame people or judge anyone.
> >>>
> >>>So the goal of my e-mail is to open the discussion and have a *real*
> >>>collaboration between Fuel team which seems to have a lot of good Puppet
> >>>engineers and Puppet OpenStack team.
> >>>
> >>>We had this kind of discussion at the Summit (in Vancouver and also
> >>>Paris, and even before). Now I would like to officialy know if you are
> >>>interested or not to be more involved.
> >>>I'm also open at any feedback about Puppet OpenStack group and if
> >>>something blocks you to contribute more.
> >>>
> >>>We have the same goals, having Puppet modules better. I think it can be
> >>>win/win: you have less diff with upstream and we have more hands in our
> >>>module maintenance.
> >>>Thank you for reading so far, and I'm looking forward to reading from you.