Open Stack

On Sep 4, 2014, at 3:24 AM, Daniel P. Berrange <berrange at redhat.com> wrote:

> Position statement
> ==================
> 
> Over the past year I've increasingly come to the conclusion that
> Nova is heading for (or probably already at) a major crisis. If
> steps are not taken to avert this, the project is likely to loose
> a non-trivial amount of talent, both regular code contributors and
> core team members. That includes myself. This is not good for
> Nova's long term health and so should be of concern to anyone
> involved in Nova and OpenStack.
> 
> For those who don't want to read the whole mail, the executive
> summary is that the nova-core team is an unfixable bottleneck
> in our development process with our current project structure.
> The only way I see to remove the bottleneck is to split the virt
> drivers out of tree and let them all have their own core teams
> in their area of code, leaving current nova core to focus on
> all the common code outside the virt driver impls. I, now, none
> the less urge people to read the whole mail.

I am highly in favor of this approach (and have been for at
least a year). Every time we have brought this up in the past
there has been concern about the shared code, but we have to
make a change. We have tried various other approaches and none
of them have made a dent.

+1000

Vish
> 
> 
> Background information
> ======================
> 
> I see many factors coming together to form the crisis
> 
> - Burn out of core team members from over work 
> - Difficulty bringing new talent into the core team
> - Long delay in getting code reviewed & merged
> - Marginalization of code areas which aren't popular
> - Increasing size of nova code through new drivers
> - Exclusion of developers without corporate backing
> 
> Each item on their own may not seem too bad, but combined they
> add up to a big problem.
> 
> Core team burn out
> ------------------
> 
> Having been involved in Nova for several dev cycles now, it is clear
> that the backlog of code up for review never goes away. Even
> intensive code review efforts at various points in the dev cycle
> makes only a small impact on the backlog. This has a pretty
> significant impact on core team members, as their work is never
> done. At best, the dial is sometimes set to 10, instead of 11.
> 
> Many people, myself included, have built tools to help deal with
> the reviews in a more efficient manner than plain gerrit allows
> for. These certainly help, but they can't ever solve the problem
> on their own - just make it slightly more bearable. And this is
> not even considering that core team members might have useful
> contributions to make in ways beyond just code review. Ultimately
> the workload is just too high to sustain the levels of review
> required, so core team members will eventually burn out (as they
> have done many times already).
> 
> Even if one person attempts to take the initiative to heavily
> invest in review of certain features it is often to no avail.
> Unless a second dedicated core reviewer can be found to 'tag
> team' it is hard for one person to make a difference. The end
> result is that a patch is +2d and then sits idle for weeks or
> more until a merge conflict requires it to be reposted at which
> point even that one +2 is lost. This is a pretty demotivating
> outcome for both reviewers & the patch contributor.
> 
> 
> New core team talent
> --------------------
> 
> It can't escape attention that the Nova core team does not grow
> in size very often. When Nova was younger and its code base was
> smaller, it was easier for contributors to get onto core because
> the base level of knowledge required was that much smaller. To
> get onto core today requires a major investment in learning Nova
> over a year or more. Even people who potentially have the latent
> skills may not have the time available to invest in learning the
> entire of Nova.
> 
> With the number of reviews proposed to Nova, the core team should
> probably be at least double its current size[1]. There is plenty of
> expertize in the project as a whole but it is typically focused
> into specific areas of the codebase. There is nowhere we can find
> 20 more people with broad knowledge of the codebase who could be
> promoted even over the next year, let alone today. This is ignoring
> that many existing members of core are relatively inactive due to
> burnout and so need replacing. That means we really need another
> 25-30 people for core. That's not going to happen.
> 
> 
> Code review delays
> ------------------
> 
> The obvious result of having too much work for too few reviewers
> is that code contributors face major delays in getting their work
> reviewed and merged. From personal experience, during Juno, I've
> probably spent 1 week in aggregate on actual code development vs
> 8 weeks on waiting on code review. You have to constantly be on
> alert for review comments because unless you can respond quickly
> (and repost) while you still have the attention of the reviewer,
> they may not be look again for days/weeks.
> 
> The length of time to get work merged serves as a demotivator to
> actually do work in the first place. I've personally avoided doing
> alot of code refactoring & cleanup work that would improve the
> maintainability of the libvirt driver in the long term, because
> I can't face the battle to get it reviewed & merged. Other people
> have told me much the same. It is not uncommon to see changes that
> have been pending for 2 dev cycles, not because the code was bad
> but because they couldn't get people to review it. Contributors
> will simply walk away from nova if that happens too often.
> 
> Even when fate is on your side and code is reviewed, the chances
> of it getting a success result from the CI systems first time
> around is slim due to false failures. This really compounds the
> already poor experiance of submitting code to Nova.
> 
> 
> Marginalization of areas
> ------------------------
> 
> Since the core team has far more work to do than it can manage, it
> has to prioritize what it looks at. The core team figures out what
> the overall project priorities are and will focus more effort in
> to those areas. Individual members will also focus their attention
> in areas where they have personal interest. Unfortunately the core
> team is not representative of the entire of Nova codebase. The
> inevitable result is that the HyperV and VMWare drivers can often
> loose out in the battle for attention. In the past we've said that
> it is the responsibility of people in those teams to invest in
> learning the entire of Nova so that they have the knowledge required
> to be promoted to core. I used to support that approach, but now
> consider to be flawed due to the increased difficulty of *anyone*
> getting onto core. The time investment required is simply too great
> to expect people to undertake it. The marginalized areas have no
> freedom to self-organize to solve their own problems because they
> are forever dependant on the core team bottleneck.
> 
> 
> Increasing size
> ---------------
> 
> There is a long standing policy that the Nova virt driver API is
> considered unstable and thus all virt driver implementations should
> ultimately be part of the Nova codebase. In Juno it is likely that
> the Ironic driver will be merged into Nova. In a future release we
> may yet see the Docker driver return to the Nova tree.
> 
> The result of merging yet more drivers is that there will be yet
> more work for nova reviewers to do. It is far from obvious that
> merging new drivers will be accompanied by new members on the core
> team. So it is likely that the workload is going to get worse over
> future releases.
> 
> Splitting out the scheduler will be beneficial in reducing the
> review backlog, but probably not enough to counter the growth from
> virt drivers. Killing of nova-network is unlikely to help at all,
> since that consumes little-to-no review time currently [2]. 
> 
> 
> Exclusion of non-corporate devs
> -------------------------------
> 
> There is a strong push from nova core for everything that is merged
> into Nova to be accompanied by CI testing. This certainly makes sense
> from the POV of overall product quality and reducing the burden on
> the core reviewers to catch all mistakes through code review. What
> we don't take into account is that setting up and maintaining such
> testing infrastructure requires a major investment in terms of both
> hardware costs and man power. It has already been seen that this is
> too much to bear for some companies who contribute to Nova, eg with
> the Docker driver [3]. Developers who are not affiliated with any
> company do not stand any realistic chance of meeting the CI testing
> needs unless they're lucky that their feature can be covered by an
> existing running CI system. This looks like it could effectively
> prevent support for a community submitted FreeBSD BHyve driver from
> being merged, no matter how useful it might be to users who want it.
> NB, now a FreeBSD BHyve driver would probably be done as part of the
> libvirt driver, which complicates this particular point I'm trying
> to make, since I don't suggest reducing testing of the libvirt driver
> compared to what it has today.
> 
> I don't want to get into a detailed testing discussion here really,
> since that's somewhat of a tangent to the question of our dev and
> review process. I am, however, concerned when our testing policy
> forces maintainers of some virt drivers into the position of being
> treated as second class citizens within the project as a whole, with
> a different development structure to the in-tree approved drivers.
> That said, Docker probably benefits from being out of tree, since it
> thus avoids the painful nova core bottleneck entirely.
> 
> 
> Problem summary
> ---------------
> 
> The common thread through most of these problems is that the nova
> core team is a massive bottleneck in the development process.
> Processes adopted (or under discussion) by the core team are
> fundamentally not helping to remove the bottleneck. Rather they are
> introducing new layers of beaurocracy so that we can feel justified
> in telling contributors that we are going to ignore or reject their
> work. At best this is going to result in far less useful work taking
> place in Nova. At worst this is further reducing the ability of
> people to self organize to solve the problems, will cause our
> contribtors to leave the community and possibly even force some virt
> drivers to go out of tree to get their work done. Death by a thousand
> cuts.
> 
> A sub-thread is around the idea that our current structure of one big
> repo also has other negative consequences for drivers who may not be
> able to meet the same high standards as the rest of the drivers. A
> driver is either in or out of the club, and if its out of the club
> life is made comparatively harder for its developers & users. By all
> means have rules around that requirements for a release to use the
> openstack trademarks based on CI testing coverage, but don't let that
> penalize the actual development process itself.
> 
> Overall Nova is being increasingly hostile to its community of
> contributors. I don't mean this as a result of any sense of malice
> or ill-will. What we're seeing is merely a symptom of a hard worked
> team struggling to survive with a burden they can no longer be
> reasonably expected to cope with. Nova core has done an amazing job
> at surviving for so long as the project grew much larger & more
> quickly than anyone probably expected. The time has come for some
> radical changes to let nova adapt & evolve to the next level.
> 
> This is a crisis. A large crisis. In fact, if you got a moment, it's
> a twelve-storey crisis with a magnificent entrance hall, carpeting
> throughout, 24-hour portage, and an enormous sign on the roof,
> saying 'This Is a Large Crisis'. A large crisis requires a large
> plan.
> 
> 
> Proposal / solution
> ===================
> 
> In the past Nova has spun out its volume layer to form the cinder
> project. The Neutron project started as an attempt to solve the
> networking space, and ultimately replace the nova-network. It
> is likely that the schedular will be spun out to a separate project.
> 
> Now Neutron itself has grown so large and successful that it is
> considering going one step further and spinning its actual drivers
> out of tree into standalone add-on projects [4]. I've heard on the
> grapevine that Ironic is considering similar steps for hardware
> drivers.
> 
> The radical (?) solution to the nova core team bottleneck is thus to
> follow this lead and split the nova virt drivers out into separate
> projects and delegate their maintainence to new dedicated teams.
> 
> - Nova becomes the home for the public APIs, RPC system, database
>   persistent and the glue that ties all this together with the
>   virt driver API.
> 
> - Each virt driver project gets its own core team and is responsible
>   for dealing with review, merge & release of their codebase.
> 
> Note, I really do mean *all* virt drivers should be separate. I do
> not want to see some virt drivers split out and others remain in tree
> because I feel that signifies that the out of tree ones are second
> class citizens. It is important to set up our dev structure so that
> every virt driver is treated equally & so has equal chance to achieve
> success. As long as one driver remains in tree there will always be
> pressure for others to join it, which is exactly what we're trying
> to get away from here. By everyone being out of tree, drivers (like
> Docker) can take a decision about whether it is the right time for
> them to be investing in gating CI systems, without being penalized
> in their dev process if they make a decision to not have gate tests
> right now.
> 
> This has quite a few implications for the way development would
> operate.
> 
> - The Nova core team at least, would be voluntarily giving up a big
>   amount of responsibility over the evolution of virt drivers. Due
>   to human nature, people are not good at giving up power, so this
>   may be painful to swallow. Realistically current nova core are
>   not experts in most of the virt drivers to start with, and more
>   important we clearly do not have sufficient time to do a good job
>   of review with everything submitted. Much of the current need
>   for core review of virt drivers is to prevent the mis-use of a
>   poorly defined virt driver API...which can be mitigated - See
>   later point(s)
> 
> - Nova core would/should not have automatic +2 over the virt driver
>   repositories since it is unreasonable to assume they have the
>   suitable domain knowledge for all virt drivers out there. People
>   would of course be able to be members of multiple core teams. For
>   example John G would naturally be nova-core and nova-xen-core. I
>   would aim for nova-core and nova-libvirt-core, and so on. I do not
>   want any +2 responsibility over VMWare/HyperV/Docker drivers since
>   they're not my area of expertize - I only look at them today because
>   they have no other nova-core representation.
> 
> - Not sure if it implies the Nova PTL would be solely focused on
>   Nova common. eg would there continue to be one PTL over all virt
>   driver implementation projects, or would each project have its
>   own PTL. Maybe this is irrelevant if a Czars approach is chosen
>   by virt driver projects for their work. I'd be inclined to say
>   that a single PTL should stay as a figurehead to represent all
>   the virt driver projects, acting as a point of contact to ensure
>   we keep communication / co-operation between the drivers in sync.
> 
> - A fairly significant amount of nova code would need to be
>   considered semi-stable API. Certainly everything under nova/virt
>   and any object which is passed in/out of the virt driver API.
>   Changes to such APIs would have to be done in a backwards
>   compatible manner, since it is no longer possible to lock-step
>   change all the virt driver impls. In some ways I think this would
>   be a good thing as it will encourage people to put more thought
>   into the long term maintainability of nova internal code instead
>   of relying on being able to rip it apart later, at will.
> 
> - The nova/virt/driver.py class would need to be much better
>   specified. All parameters / return values which are opaque dicts
>   must be replaced with objects + attributes. Completion of the
>   objectification work is mandatory, so there is cleaner separation
>   between virt driver impls & the rest of Nova.
> 
> - If changes are required to common code, the virt driver developer
>   would first have to get the necccessary pieces merged into Nova
>   common. Then the follow up virt driver specific changes could be
>   proposed to their repo. This implies that some changes to virt
>   drivers will still contend for resource in the common nova repo 
>   and team. This contention should be lower than it is today though
>   since the current nova core team should have less code to look 
>   after per-person on aggregate.
> 
> - Changes submitted to nova common code would trigger running of CI
>   tests against the external virt drivers. Each virt driver core team
>   would decide whether they want their driver to be tested upon Nova
>   common changes. Expect that all would choose to be included to the
>   same extent that they are today. So level of validation of nova code
>   would remain at least at current level. I don't want to reduce the
>   amount of code testing here since that's contrary to the direction
>   we're taking wrt testing.
> 
> - Changes submitted to virt drivers would trigger running CI tests
>   that are applicable. eg changes to libvirt driver repo would not
>   involve running database migration tests, since all database code
>   is isolated in nova. libvirt changes would not trigger vmware,
>   xenserver, ironic, etc CI systems. Virt driver changes should
>   see fewer false positives in the tests as a result, and those
>   that do occur should be more explicitly related to the code being
>   proposed. eg a change to vmware is not going to trigger a tempest
>   run that uses libvirt, so non-deterministic failures in libvirt
>   will no longer plague vmware developers reviews. This would also
>   make it possible for VMWare CI to be made gating for changes to
>   the VMWare virt driver repository, without negatively impacting
>   other virt drivers. So this change should increase testing quality
>   for non-libvirt virt drivers and reduce pain of false failures
>   for everyone.
> 
> - Virt drivers shouldn't use oslo incubator code from nova, since
>   that can be replaced any time and isn't upgrade safe. Ideally most
>   of the incubator stuff virt drivers need should turn into stable
>   oslo APIs. Failing that, virt drivers would need their own copy
>   of the incubated code in their module namespace, to avoid clash
>   or the need to lock-step upgrade code across separate git repos.
> 
> Overall the outcome is that
> 
> - Far larger pool of people able to approve changes for merge
>   across nova core and the virt driver core teams.
> 
> - Faster review & merge for virt driver patches that don't involve
>   changes to common nova code, with less CI system testing pain.
> 
> - Ability to set priority of work in virt drivers without a 3rd
>   party being a bottleneck, where the work doesn't involve changes
>   to common nova code.
> 
> - Each virt driver team can accept as many features as they feel
>   able to deal with, without it negatively impacting amount of
>   features that other virt driver teams can accept.
> 
> - Virt drivers have flexibility to set their own policies on testing
>   without being penalized in the way they then develop their code.
> 
> 
> The migration
> -------------
> 
> Obviously a proposal such as this is a pretty major undertaking. It
> should be clear that it could not be done in a short amount of time.
> It is suggested that it be phased in over two dev cycles. In the Kilo
> release the focus would be on prep work:
> 
>  - Formalizing the separation between the virt driver impls and the
>    rest of the nova codebase. Figure out exactly which areas of 
>    Nova internal code will need to be marked as 'semi-stable' for 
>    use by virt drivers, and ensure their APIs are sufficiently
>    future proof.
> 
>  - Discussions with the infrastructure, docs, release, etc teams to
>    identify impacts on them and do any required prep work.
> 
>  - Identify the teams which will lead the new virt driver projects.
>    eg core reviewers, PTL or Czars for each job if applicable
> 
>  - Probably more things I can't think of right now
> 
> Then at the start of the Lxxxx release, the virt drivers would
> actually be split out into separate git repos and start their dev
> process for the future. So for bulk of Lxxxx the drivers would be
> on their own. The two Lxxxx rc milestones would allow us to ensure
> our release processes were working well with the split drivers
> before the Lxxxx final release.
> 
> 
> Final thought
> -------------
> 
> Overall consider this a vote of no confidence in nova continuing to
> operate as it does today. As mentioned above this is not intended to
> be disrepectful to the effort every nova core member has put in, just
> a reflection on the changed environment we find ourselves in. Fiddling
> with our processes for the prioritization of work cannot fix the
> fundamental fact that nova core today is a massive single point of
> failure & bottleneck, increasingly crippling the project. The only way
> to address this is by a radical re-organization of our project to
> remove the bottlenecks by modularization of the project & leaders.
> Keeping a single team and adding more/changing process is simply akin
> to shifting deckchairs on the titanic and not a viable option to coninue
> with long term.
> 
> Now, I'm realistic. Even with every driver separated out, I expect
> that each of them will individually still have more work proposed
> than their respective core teams have time to review. The new structure
> will, however, make it easier for the core individal teams to grow &
> adapt in ways that suit their specific needs. For self-contained virt
> driver changes it will mean that acceptance of work by one team will
> not take away capacity from another team. Further the burden of
> knowledge required to make it onto a virt driver core team would be
> greatly reduced due to the narrower focus of each core team, so we'll
> be able to promote good talent onto virt driver core teams more quickly.
> 
> Thanks for reading so far. Now lets make some real change to prepare
> us for future sustainability & even growth.
> 
> Regards,
> Daniel
> 
> [1] http://lists.openstack.org/pipermail/openstack-dev/2014-August/044459.html
> [2] There was a ban on changes to nova-network for much of the past two
>    cycles. It was relaxed primarily to allow full conversion of nova
>    codebase to use objects, not for major new feature development.
> [3] http://lists.openstack.org/pipermail/openstack-dev/2014-July/040443.html
> [4] http://lists.openstack.org/pipermail/openstack-dev/2014-August/043036.html
> 
> -- 
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-manager.org :|
> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140910/a25b7586/attachment.pgp>