Open Stack

On Thu, 2014-09-11 at 07:36 -0400, Sean Dague wrote:
> >>> b) The conflict Dan is speaking of is around the current situation where we
> >>> have a limited core review team bandwidth and we have to pick and choose
> >>> which virt driver-specific features we will review. This leads to bad
> >>> feelings and conflict.
> >>
> >> The way this worked in the past is we had cores who were subject
> >> matter experts in various parts of the code -- there is a clear set of
> >> cores who "get" xen or libivrt for example and I feel like those
> >> drivers get reasonable review times. What's happened though is that
> >> we've added a bunch of drivers without adding subject matter experts
> >> to core to cover those drivers. Those newer drivers therefore have a
> >> harder time getting things reviewed and approved.
> > 
> > FYI, for Juno at least I really don't consider that even the libvirt
> > driver got acceptable review times in any sense. The pain of waiting
> > for reviews in libvirt code I've submitted this cycle is what prompted
> > me to start this thread. All the virt drivers are suffering way more
> > than they should be, but those without core team representation suffer
> > to an even greater degree.  And this is ignoring the point Jay & I
> > were making about how the use of a single team means that there is
> > always contention for feature approval, so much work gets cut right
> > at the start even if maintainers of that area felt it was valuable
> > and worth taking.
> 
> I continue to not understand how N non overlapping teams makes this any
> better. You have to pay the integration cost somewhere. Right now we're
> trying to pay it 1 patch at a time. This model means the integration
> units get much bigger, and with less common ground.

OK, so look at a concrete example: in 2002, the Linux kernel went with
bitkeeper precisely because we'd reached the scaling limit of a single
integration point, so we took the kernel from a single contributing team
to a bunch of them.  This was expanded with git in 2005 and leads to the
hundreds of contributing teams we have today.

The reason this scales nicely is precisely because the integration costs
are lower.  However, there are a couple of principles that really assist
us getting there.  The first is internal API management: an Internal API
is a contract between two teams (may be more, but usually two).  If
someone wants to change this API they have to negotiate between the two
(or more) teams.  This naturally means that only the affected components
review this API change, but *only* they need to review it, so it doesn't
bubble up to the whole kernel community.  The second is automation:
linux-next and the zero day test programme build and smoke test an
integration of all our development trees.  If one team does something
that impacts another in their development tree, this system gives us
immediate warning.  Basically we run continuous integration, so when
Linus does his actual integration pull, everything goes smoothly (that's
how we integrate all the 300 or so trees for a kernel release in about
ten days).  We also now have a lot of review automation (checkpatch.pl
for instance), but that's independent of the number of teams

In this model the scaling comes from the local reviews and integration.
The more teams the greater the scaling.  The factor which obstructs
scaling is the internal API ... it usually doesn't make sense to
separate a component where there's no API between the two pieces ...
however, if you think there should be, separating and telling the teams
to figure it out is a great way to generate the API.   The point here is
that since an API is a contract, forcing people to negotiate and abide
by the contract tends to make them think much more carefully about it.
Internal API moves from being a global issue to being a local one.

By the way, the extra link work is actually time well spent because it
means the link APIs are negotiated by teams with use cases not just
designed by abstract architecture.  The greater the link pain the
greater the indication that there's an API problem and the greater the
pressure on the teams either end to fix it.  Once the link pain is
minimised, the API is likely a good one.

> Look at how much active work in crossing core teams we've had to do to
> make any real progress on the neutron replacing nova-network front. And
> how slow that process is. I think you'll see that hugely show up here.

Well, as I said, separating the components leads to API negotiation
between the teams  Because of the API negotiation, taking one thing and
making it two does cause more work, and it's visible work because the
two new teams get to do the API negotiation which didn't exist before.
The trick to getting the model to scale is the network effect.  The
scaling comes by splitting out into high numbers of teams (say N) the
added work comes in the links (the API contracts) between the N teams.
If the network is star shaped (everything touches everything else), then
you've achieved nothing other than a large increase in work because you
now have N(N-1) links to negotiate and they're global not local.
However, if the connections are nicely local and hierarchical, you find
that the number of connections is much lower than N^2 (in fact o(N) is
the ideal scaling ratio because that means the network is largely local)
and you gain a lot of scaling.

The point is just separating may not give scale because you also need to
minimise the number of links by seeking locality.  In the nova case,
separating nova core from drivers core will give you a link, but the API
is fairly well known already.  Separating individual drivers from driver
core gives you one link per driver, but no link to the core, so if you
make the drivers core team up from the drivers teams, you should give
added velocity to driver development and review and it only needs to
negotiate with the nova core if the driver core API needs modifying.
The drivers can negotiate with the drivers core for their local needs.
The pain on the nova<->driver core link is what keeps that API sane
because changes have to be well considered.

James