[openstack-dev] [Nova] Frustrations with review wait times

Joe Gordon joe.gordon0 at gmail.com
Wed Aug 28 13:58:48 UTC 2013

On Aug 28, 2013 9:42 AM, "Shawn Hartsock" <hartsocks at vmware.com> wrote:
> tl;dr at the end... I can ramble on a bit.
> I agree with Daniel.
> I'm not a core reviewer, but I'm trying to think like one. Over the last
few weeks I've divested myself of almost all coding tasks, instead trying
to increase the size of the community that is actively contributing to my
area of expertise. I have indeed gone batty! I've caught myself a few
times, and the frustration of feeling like I couldn't contribute code even
if I wanted to is getting to be a bit too much.
> The common refrain I've heard is, "That's OpenSource" as if this is a
natural state of affairs for OpenSource projects. I've been either on or
around OpenSource projects for nearly 20 years at this point and I really
feel this doesn't have to be the case. Any project is doomed to have an
internal structure that mirrors the organization that maintains it. That
means software beyond a certain scale becomes part engineering and part
> In OpenSource projects that I have worked on recently, the way scale was
handled was to break up the project into pieces small enough for teams of 1
to 5 to handle. The core framework developers worked on exposing API to the
plugin developers. Each plugin developer would then focus on how their
plugin could expose both additional API and leverage framework API.
Feedback went from the application developers to the plugin developers and
up to the core developers. This whole divide-and-conquer strategy was aided
by the fact that we could lean heavily on a custom dependency management
and code/binary distribution system leveraged inside the framework itself.
It meant that package structure and distribution could be controlled by the
community directly to suit its needs. That makes a powerful combination for
building a flexible system but requires a fair amount of infrastructure in
code and hardware.
> It wasn't a perfect solution. This strategy meant that an application or
deployment became the coordination of plugins mixed at run-time. While
efforts were made to test common combinations, it was impossible to test
all combinations. That often meant people in the field were using
combinations that nobody on official teams had ever considered. Because the
plugins weren't on the same release cycle as the core framework (and even
in different code repositories and release infrastructures) a plugin could
release weekly or once every few years depending on its needs and
> There is a separate dysfunction you'll see if you go down this path. Core
API must necessarily lead plugin implementation ... which means you
sometimes get nonsense API with no backing. To solve this a few plugins are
deemed "core" plugins and march in-step with the API release cycle. Then
there's the added burden of longer backward compatibility cycles that
necessarily stretch longer and longer leaving deprecated API lying around
for years as plugin developers are coaxed into leaving them behind (and
subsequent plugin users are coaxed to upgrade). Some things slow down while
others speed up. The core API's evolution slows, the plugin/driver speeds
up. Is that a fair trade off? It's a judgement call. No right answer.
> In the end you trade one kind of problem for another and one kind of
coordination for another. There's no clean answer that just works for this
kind of problem and its why we have so many different kinds of governments
in the world. Because, ultimately, that's what human coordination becomes
if you don't watch out and "one size does not fit all".
> Based on my experiences this last cycle, I think nova is pretty well
broken down already. Each driver is practically its own little group, the
scheduler is surprisingly well fenced off, as are cells. As for our
sub-team I think we could have moved much faster if we had been able to
approve our own driver blueprints some of which have been in review since
May and others which have 30+ revisions updated every few days hoping for
attention. It's part of why I moved to watching over people's work instead
of doing my own and I now spend most of my time giving feedback to reviews
other people are working on and seeking out expert opinions on other
people's efforts.

Updating patches for attention often does the opposite for me.

When I see a patch set being revised every few days, that makes me think (
perhaps incorrectly) that the patch is still in active development and I am
inclined to review something else.

On a related note, I really like when the developer adds a gerrit comment
saying why the revision, that makes my life as a reviewer easier.

> It's not a pleasant place to be and every time I pick up something to
work on I either get pulled away or someone else picks up the job and
finishes before I can even get started. I imagine this is much like what it
is to be a core developer and that this contest of interest is the same
strain the core-reviewers feel. You end up picking your own work and
neglecting others or falling on the sword so other people can do their work
and doing none of your own. Frankly, I don't want to use this strategy next
cycle because it is far too unsatisfying for me.
> BTW:
> Anyone interested in this on an academic level, most of these ideas I
have are from vague recollections of college readings of the work of W.
Edwards Deming, Coase theorem, and more humorously and directly: Conway's
law. But, I don't do that for a living so I may not understand it too well.
Mostly, I was interested in how to coordinate large numbers of developers.
> tl;dr please don't create a reviewer caste (it's not fun), break the
project into pieces, give the pieces autonomy... for me, that tends to be
more fun.
> # Shawn Hartsock
> ----- Original Message -----
> > From: "Robert Collins" <robertc at robertcollins.net>
> > To: "Daniel P. Berrange" <berrange at redhat.com>
> > Cc: "OpenStack Development Mailing List" <
openstack-dev at lists.openstack.org>
> > Sent: Wednesday, August 28, 2013 6:51:22 AM
> > Subject: Re: [openstack-dev] [Nova] Frustrations with review wait times
> >
> > On 28 August 2013 22:39, Daniel P. Berrange <berrange at redhat.com> wrote:
> >
> > > No, IIUC, Joshua was suggesting that core team members spend one cycle
> > > doing reviews only, with no coding, and then reverse for the next
> > > That is just far too coarse/crude. Core team members need to be free
> > > balance their time between reviews and coding work on an ongoing
> > > just as any other member of the community can.
> >
> > Oh! Yes, thats way too heavy.
> >
> > I do wonder about tweaking the balance more [or scaling the review
> > team :)], but only-reviews would drive anyone batty.
> >
> > -Rob
> >
> > --
> > Robert Collins <rbtcollins at hp.com>
> > Distinguished Technologist
> > HP Converged Cloud
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130828/da4baed9/attachment.html>

More information about the OpenStack-dev mailing list