Open Stack

On Thu, Sep 04, 2014 at 06:48:33PM -0400, Russell Bryant wrote:
> On 09/04/2014 06:24 AM, Daniel P. Berrange wrote:
> > Position statement
> > ==================
> > 
> > Over the past year I've increasingly come to the conclusion that
> > Nova is heading for (or probably already at) a major crisis. If
> > steps are not taken to avert this, the project is likely to loose
> > a non-trivial amount of talent, both regular code contributors and
> > core team members. That includes myself. This is not good for
> > Nova's long term health and so should be of concern to anyone
> > involved in Nova and OpenStack.
> > 
> > For those who don't want to read the whole mail, the executive
> > summary is that the nova-core team is an unfixable bottleneck
> > in our development process with our current project structure.
> > The only way I see to remove the bottleneck is to split the virt
> > drivers out of tree and let them all have their own core teams
> > in their area of code, leaving current nova core to focus on
> > all the common code outside the virt driver impls. I, now, none
> > the less urge people to read the whole mail.
> 
> Fantastic write-up.  I can't +1 enough the problem statement, which I
> think you've done a nice job of framing.  We've taken steps to try to
> improve this, but none of them have been big enough.  I feel we've
> reached a tipping point.  I think many others do too, and several
> proposals being discussed all seem rooted in this same core issue.
> 
> When it comes to the proposed solution, I'm +1 on that too, but part of
> that is that it's hard for me to ignore the limitations placed on us by
> our current review infrastructure (gerrit).
> 
> If we ignored gerrit for a moment, is rapid increase in splitting out
> components the ideal workflow?  Would we be better off finding a way to
> finally just implement a model more like the Linux kernel with
> sub-system maintainers and pull requests to a top-level tree?  Maybe.
> I'm not convinced that split of repos is obviously better.
> 
> You make some good arguments for why splitting has other benefits.

For a long time I've use the LKML 'subsystem maintainers' model as the
reference point for ideas. In a more LKML like model, each virt team
(or other subsystem team) would have their own separate GIT repo with
a complete Nova codebase, where they did they day to day code submissions,
reviews and merges. Periodically the primary subsystem maintainer would
submit a large pull / merge requests to the overall Nova maintainer.
The $1,000,000 question in such a model is what kind of code review
happens during the big pull requests to integrate subsystem trees. 

The closest example I can see is what's happening with the Ironic
driver merge reviews. I'm personally finding review of that to be
quite a burdensome activity, because all comments on the merge
review then get fed back to the orginal maintainers who do a new
round of patch + review in Ironic tree and then we get a new version
submitted back to nova tree for merge. Rinse, repeat.

So my biggest fear with a model where each team had their own full
Nova tree and did large pull requests, is that we'd suffer major
pain during the merging of large pull requests, especially if any
of the merges touched common code. It could make the pull requests
take a really long time to get accepted into the primary repo.

By constrast with split out git repos per virt driver code, we will
only ever have 1 stage of code review for each patch. Changes to
common code would go straight to main nova common repo and so get
reviewed by the experts there without delay, avoiding the 2nd stage
of review from merge requests.

The more I think abut this, the more attracted I am to the idea
that separate repos will facilitate us doing more targetted testing
and allow 3rd party CI to become gating over their respective virt
driver codebases.

Finally the LKML model would still leave some drivers at a disadvantage
for development, if they're not able to meet the standards we require
in terms of CI testing, to be accepted into the primary repo.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|