Open Stack

On Thu, Jun 11, 2015 at 05:39:28PM -0400, Emilien Macchi wrote:
> On 06/11/2015 10:36 AM, Matthew Mosesohn wrote:
> > I'm not saying it's the most community-oriented approach, but Fuel
> > would have never evolved and matured without it. The attribution in
> > commits is lost because our directory namespace differs such that it
> > can't just be merged cleanly. Revisiting submodules is an option,
> > but it led to maintenance problems in the past.
> What kind of problems?

It was before I got involved with Fuel, but I can offer a guess.
Managing submodules introduces operational overhead and with it more
opportunities for failures and mishaps than a single flat git repo.

Flat repo makes it more difficult to reconcile with upstream modules,
but, when using the process I have described in my previous email to
this thread, not as difficult as one could think. We also reconcile an
average module with upstream much less frequently (no more than once per
Fuel release) than we make commits to that module (many times per
release), which also tilts the overhead balance if favor of using a flat
repo.

> You also could have used forks of modules, applied your patches and done
> rebase from time to time when you like.
> Using a Puppetfile in your installer and you're all set.

We do the "apply patches and rebase from time to time" part in a single
repo, operating on a subdirectly within it doesn't actually add any
overhead to this part of the workflow, as long as we keep track of sync
commits properly.

> The "maintenance problems" thing does not sound right to me here, I
> think it's more expensive to maintain files than git repositories.

To sum up, I don't think the difference is that great, or impact of
doing it one way or the other that important. The current layout works
well enough for Fuel team, and as you said yourself, developers of
upstream modules are not likely to pro-actively harvest Fuel for
bugfixes even if we change our repository structure.

> > The difference is that Fuel is being worked on in the open in
> > StackForge. Anyone is free to contribute to Fuel as he or she
> > wishes, take our patches, or review changesets.
> Should not it be the way around?
> Puppet OpenStack modules provide the original code. If there is a bug,
> it has to be fixed in the modules. Puppet OpenStack developers don't
> have time/bandwidth and moreover don't want to periodically have a
> look at Fuel git history. I'm not sure this is the best solution for
> the community.
(...)
> The reality (and again I won't blame any patch, you can find them on
> Gerrit) is that most of patches are not merged and in staled status.
> If I can suggest something, the policy should be more "upstream first"
> where you submit a patch upstream, you backport it downstream, and in
> the until it's merged you should make sure it land upstream after
> community review process. The last step is I think the problem I'm
> mentioning here and part of the root cause of this topic.

Yes, this right here is the biggest point of contention in the whole
discussion.

The most problematic implication of what you're asking for is the
additional effort that it would require from Fuel developers. When you
say that Puppet OpenStack developers don't have time to look at Fuel git
history for bugfixes, and then observe that actually Fuel developers do
propose their patches to upstream, but those patches are stalled in the
community review process, this indicates that you don't consider taking
over and landing these patches a priority:

http://lifehacker.com/5892948/instead-of-saying-i-dont-have-time-say-its-not-a-priority

The fact that you have started this thread means that you actually do
care to get these bugfixes into Puppet OpenStack. If that's true, maybe
you can consider a middleground approach: Fuel team agrees to propose
all our changes to upstream (i.e. do a better job at something we've
already committed to unilaterally), and you help us land the patches we
propose, and take over those that get stalled when the submitter from
Fuel team has moved on to other tasks?

A better alternative would be to make all upstream Puppet OpenStack
directly usable in Fuel, but even if we figure out a way to make that
work, it will take a long journey to get there. On the upstream side,
Fuel core reviewers would have to also be upstream core reviewers, and
Fuel CI would have to be allowed to vote on upstream commits. On Fuel
side, we'd have to complete the reconciliation and modularization of all
our forked modules, and move all Fuel CI to openstack-infra. The
potential benefits for both sides are tremendous, but only after we
essentially merge the two projects. Even if that's achievable, is that
something whole Puppet OpenStack community is interested in?

> Again, I'm not sure this is the right direction. The official channel
> for Puppet OpenStack modules is #puppet-openstack and this is the place
> to be when you're involved in the Puppet OpenStack community.
> I would suggest to rewrite this thing in "if a bug is discovered in
> Puppet OpenStack after it's been reported or fixed in Fuel, then folks
> (from both groups) should collaborate on Puppet OpenStack official
> channel to fix it upstream".
> IMHO, Fuel IRC channel should relate to Fuel specific things.
> 
> As a example, RDO has #rdo-puppet talking about Puppet OpenStack
> downstream (packstack, etc) things and use #puppet-openstack for
> upstream related bugs. I think this is the way to go if we want to
> improve our collaboration.

Agreed, Fuel developers should use #puppet-openstack to discuss upstream
work, expecting upstream developers to come to #fuel-dev is not
reasonable. As discussed above, the trick is to make sure that Fuel
developers can afford to prioritize the upstream work. Without that,
collaboration won't happen on any IRC channel.

> > If you have specific bugs or commits you'd like to discuss, let's work
> > them out. I believe I can get the Fuel Library team to agree to do a
> > walk through all the open bugs in Puppet OpenStack and see if we have
> > any related fixes or bug reports.
> 
> Like I already said, I won't share any patch/bug because I don't want to
> blame anyone here, this is not the goal. Going thru Launchpad and
> Gerrit, you'll easily see what I mean.

I think what Matt meant here is that even if we have a policy to
upstream our changes, people miss things, so it's possible that some of
our patches were not proposed to upstream, and it would be helpful of
you to point those cases out so that we can fix that and propose those
patches properly (referring to the relevant bugs etc.) I see how doing
that in on a public mailing list can turn into a blame game, what about
reporting an LP bug against Fuel in the form of "this patch, commit, or
line of code modifies the code forked from upstream module, please
report to upstream as per policy"?

-- 
Dmitry Borodaenko