Open Stack

On Thu, Sep 04, 2014 at 10:44:17PM -0600, John Griffith wrote:
> Just some thoughts and observations I've had regarding this topic in Cinder
> the past couple of years.  I realize this is a Nova thread so hopefully
> some of this can be applied in a more general context.
> 
> TLDR:
> 1. I think moving drivers into their own repo is just shoveling the pile to
> make a new pile (not really solving anything)

I'm not familiar with Cinder, but for Nova it would certainly have clear
benefits and not merely be shoveling the pile. Specifically it would

 - Easily let us double the number of "core" reviewers on aggregate

 - Reduce the bar for getting into a driver core team thus increasing
   the talent pool we can promote from.

 - Work accepted in a release for one driver would not reduce the
   bandwidth for another driver to accept work, since their review
   teams are separate

 - We can have more targetted testing, which will reduce the amount
   of bogus gate failures people get when submitting reviews and
   allow every driver to have gating CI jobs without impacting the
   other drivers

> 2. Removal of drivers other than the reference implementation for each
> project could be the healthiest option
>     a. Requires transparent, public, automated 3'rd party CI
>     b. Requires a TRUE plugin architecture and mentality
>     c. Requires a stable and well defined API

As mentioned in the original mail I don't want to see a situation where
we end up with some drivers in tree and others out of tree as it sets up
bad dynamics within the project. Those out of tree will always have the
impression of being second class citizens and thus there will be constant
pressure to accept drivers back into tree. The so called 'reference'
driver that stayed in tree would also continue to be penalized in the
way it is today, and so its development would be disadvantaged compared
to the out of tree drivers.

> 3. While I'm still sort of a fan of the removal of drivers, I do think
> Cinder is "making it work", there have been missteps and yes it's a pain
> sometimes but it's working "ok" and we've got plans to try and improve
> 
> 4. Adding restrictions like drivers only in first milestone and more
> intense scrutinization of features will go a long way to help resolve the
> issues we do have currently

Not in nova at least. We have a fundamental bottleneck in nova and
simply re-arranging review priorities in this kind of way will never
fix it. We've tried many different approaches to prioritization of
work and the only result is that we've got more aggressive at saying
no to contributors. This is directly resulting in the crisis we have
today.

> I've spent a fair amount of time thinking about the explosive number of
> drivers being added to Cinder over the past year or so.  I've been a pretty
> vocal proponent of the idea of removing all drivers except the LVM
> reference implementation from Cinder.  I'd rather see Vendors drivers
> maintained in their own Github Repo and truly follow a "plugin" model.
>  This of course means that Cinder has to be truly designed and maintained
> with a real plugin architecture kept in mind in every aspect of development
> (experience proves this harder to do than it sounds).  I think with things
> stable and well defined interfaces as well as 3'rd party CI this is
> actually a reasonable approach and could be effective.  I do not see how
> creating a separate repo and in essence yet another set of OpenStack
> Projects really helps with the problem.  The fact is that the biggest issue
> most people see with driver contributions is those that are made by
> organizations that work on their driver only and don't contribute back to
> the core project (whether that be in the form of reviews of core
> contributions).  I'm not sure I understand why that would be any different
> by just putting the code in a separate bucket.  In other words, getting a
> solid and consistent team working on that "project" seems like you've just
> kicked the can down the road so you don't have to deal with it.

Fundamentally people contributing to a project are doing so voluntarily
to scratch their own itch. The project leadership can help identify areas
that need work and encourage people to take up the challenge, but you
cannot force people to do the work. We've done many things in nova that
are basically inflicting a form of punishment on contributors if they
don't work on things we tell them to work on. This is not having a positive
effect, on the contrary it is resulting in alot of demovated and pissed off
contributors who are ultimately leaving the project.

I agree that splitting the virt drivers out into their own repositories is
not going to hugely help get more people to work on Nova core - that was
not the primary intention. The big focus is on unblocking development of
the virt drivers so that their contributors actually feeled their efforts
are valued by the project. If we make the project a more attractive place
to work in general that will ultimately result in more people /wanting/
to work on solving the common core issues too. Allowing the virt driver
teams to self-organize though, will also free up some resources for
dealing with core code because new teams of people will take on a portion
of the work that current nova core has todo.

> Any time I've mentioned the removal approach the response is typically that
> there's no quality control, or that Vendors won't be as willing to invest
> in OpenStack because they can focus on their own interests and get by with
> that.  The quality control one was a tough one to counter, but now that
> we're moving towards things like 3'rd party CI I'm not sure that's quite as
> significant as it was a year ago.  I'd still like to see a public record of
> testing in the form of CI, NOT just Vendor-A submitting something that
> says.. "yeah, I'm awesome".  I suspect that OpenStack adopters would look
> at things like public CI postings to determine what's worth pursuing and
> what's not.

I think quality control is a total non-issue and will in fact be helped
by splitting out the Nova drivers. We have 3rd party CI for drivers in
nova today, but none except the libvirt driver CI is gating. I don't see
us ever having 3rd party CI for vmware/hyperv/xenapi gating on the main
nova repo because we already suffer from too many false gate failures
and adding more gating jobs will make that dramatically worse. With
separate repos for each driver though, there is no problem with each
driver's CI being gating for their own repo. So this would be beneficial
for the virt drivers in terms of their testing.

> Over the past year I've become even less concerned about the issues of
> "loosing contributors".  The fact is that more an more vendors are choosing
> to strip down the driver implementation they submit to Cinder to a simple
> Interface that calls their external python library.  Some of you may
> remember a long time ago I voiced my concern that this was an AWFUL
> direction and nothing good would come of it.  Fact is, I was VERY WRONG.
>  It's actually not a bad model, vendors that have taken that approach are
> focused more heavily on Cinder than they are on their own drivers.  Really,
> IMO since a number of vendors are already half way there, maybe it's not
> that big of a deal to just go all the way.

I can certainly understand the attraction of moving most of the code into
an external python library because it frees them from the pain of dealing
with the code review & feature approval processes for a much larger portion
of the work, so allowing them to be more productive overall.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|