Open Stack

tl;dr at the end... I can ramble on a bit.

I agree with Daniel.

I'm not a core reviewer, but I'm trying to think like one. Over the last few weeks I've divested myself of almost all coding tasks, instead trying to increase the size of the community that is actively contributing to my area of expertise. I have indeed gone batty! I've caught myself a few times, and the frustration of feeling like I couldn't contribute code even if I wanted to is getting to be a bit too much.

The common refrain I've heard is, "That's OpenSource" as if this is a natural state of affairs for OpenSource projects. I've been either on or around OpenSource projects for nearly 20 years at this point and I really feel this doesn't have to be the case. Any project is doomed to have an internal structure that mirrors the organization that maintains it. That means software beyond a certain scale becomes part engineering and part state-craft.

In OpenSource projects that I have worked on recently, the way scale was handled was to break up the project into pieces small enough for teams of 1 to 5 to handle. The core framework developers worked on exposing API to the plugin developers. Each plugin developer would then focus on how their plugin could expose both additional API and leverage framework API. Feedback went from the application developers to the plugin developers and up to the core developers. This whole divide-and-conquer strategy was aided by the fact that we could lean heavily on a custom dependency management and code/binary distribution system leveraged inside the framework itself. It meant that package structure and distribution could be controlled by the community directly to suit its needs. That makes a powerful combination for building a flexible system but requires a fair amount of infrastructure in code and hardware.

It wasn't a perfect solution. This strategy meant that an application or deployment became the coordination of plugins mixed at run-time. While efforts were made to test common combinations, it was impossible to test all combinations. That often meant people in the field were using combinations that nobody on official teams had ever considered. Because the plugins weren't on the same release cycle as the core framework (and even in different code repositories and release infrastructures) a plugin could release weekly or once every few years depending on its needs and sub-community.

There is a separate dysfunction you'll see if you go down this path. Core API must necessarily lead plugin implementation ... which means you sometimes get nonsense API with no backing. To solve this a few plugins are deemed "core" plugins and march in-step with the API release cycle. Then there's the added burden of longer backward compatibility cycles that necessarily stretch longer and longer leaving deprecated API lying around for years as plugin developers are coaxed into leaving them behind (and subsequent plugin users are coaxed to upgrade). Some things slow down while others speed up. The core API's evolution slows, the plugin/driver speeds up. Is that a fair trade off? It's a judgement call. No right answer.

In the end you trade one kind of problem for another and one kind of coordination for another. There's no clean answer that just works for this kind of problem and its why we have so many different kinds of governments in the world. Because, ultimately, that's what human coordination becomes if you don't watch out and "one size does not fit all".

Based on my experiences this last cycle, I think nova is pretty well broken down already. Each driver is practically its own little group, the scheduler is surprisingly well fenced off, as are cells. As for our sub-team I think we could have moved much faster if we had been able to approve our own driver blueprints some of which have been in review since May and others which have 30+ revisions updated every few days hoping for attention. It's part of why I moved to watching over people's work instead of doing my own and I now spend most of my time giving feedback to reviews other people are working on and seeking out expert opinions on other people's efforts. 

It's not a pleasant place to be and every time I pick up something to work on I either get pulled away or someone else picks up the job and finishes before I can even get started. I imagine this is much like what it is to be a core developer and that this contest of interest is the same strain the core-reviewers feel. You end up picking your own work and neglecting others or falling on the sword so other people can do their work and doing none of your own. Frankly, I don't want to use this strategy next cycle because it is far too unsatisfying for me.

BTW:
Anyone interested in this on an academic level, most of these ideas I have are from vague recollections of college readings of the work of W. Edwards Deming, Coase theorem, and more humorously and directly: Conway's law. But, I don't do that for a living so I may not understand it too well. Mostly, I was interested in how to coordinate large numbers of developers.

tl;dr please don't create a reviewer caste (it's not fun), break the project into pieces, give the pieces autonomy... for me, that tends to be more fun.

# Shawn Hartsock

----- Original Message -----
> From: "Robert Collins" <robertc at robertcollins.net>
> To: "Daniel P. Berrange" <berrange at redhat.com>
> Cc: "OpenStack Development Mailing List" <openstack-dev at lists.openstack.org>
> Sent: Wednesday, August 28, 2013 6:51:22 AM
> Subject: Re: [openstack-dev] [Nova] Frustrations with review wait times
> 
> On 28 August 2013 22:39, Daniel P. Berrange <berrange at redhat.com> wrote:
> 
> > No, IIUC, Joshua was suggesting that core team members spend one cycle
> > doing reviews only, with no coding, and then reverse for the next cycle.
> > That is just far too coarse/crude. Core team members need to be free to
> > balance their time between reviews and coding work on an ongoing basis,
> > just as any other member of the community can.
> 
> Oh! Yes, thats way too heavy.
> 
> I do wonder about tweaking the balance more [or scaling the review
> team :)], but only-reviews would drive anyone batty.
> 
> -Rob
> 
> --
> Robert Collins <rbtcollins at hp.com>
> Distinguished Technologist
> HP Converged Cloud
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>