Open Stack

On 08/07/2014 08:54 AM, Russell Bryant wrote:
> On 08/07/2014 04:49 AM, Thierry Carrez wrote:
>> Stefano Maffulli wrote:
>>> On Wed 06 Aug 2014 02:10:23 PM PDT, Michael Still wrote:
>>>>  - we rate limit the total number of blueprints under code review at
>>>> any one time to a fixed number of "slots". I secretly prefer the term
>>>> "runway", so I am going to use that for the rest of this email. A
>>>> suggested initial number of runways was proposed at ten.
>>>
>>> oooooh, I like the 'slots/runaway model'. Sounds to me like kan ban (in 
>>> the Toyota sense not the hipster developer sense).
>>>
>>> A light in my head just went on.
>>>
>>> Let me translate what you're thinking about in other terms: the 
>>> slot/runaway model would switch what is now a push model into a "pull" 
>>> model. Currently we have patches coming in, pushed up for review. We 
>>> have then on gerrit reviewers and core reviewers shuffling through 
>>> these changesets, doing work and approve/comment. The reviewers have 
>>> little to no way to notice when they're overloaded and managers have no 
>>> way either. There is no way to identify when the process is suffering, 
>>> slowing down or not satisfying demand, if not when the backlog blows 
>>> up. As recent discussions demonstrate, this model is failing under our 
>>> growth.
>>>
>>> By switching to a model where we have a set of slots/runaway (buckets, 
>>> in Toyota's terminology) reviewers would have a clear way to *pull* new 
>>> reviews into their workstations to be processed. It's as simple as a 
>>> supermaket aisle: when there is no more pasta on the shelf, a clerk 
>>> goes in the backend and gets more pasta to refurbish the shelf. There 
>>> is no sophisticated algorithm to predict demand: it's the demand of 
>>> pasta that drives new pull requests (of pasta or changes to review).
>>>
>>> This pull mechanism would help make it very visible where the 
>>> bottlenecks are. At Toyota, for example, the amount of kanbans is the 
>>> visible way to understand the capacity of the plant. The amount of 
>>> slots/runaways would probably give us similar overview of the capacity 
>>> of each project and give us tools to solve bottlenecks before they 
>>> become emergencies.
>>
>> As an ex factory IT manager, I feel compelled to comment on that :)
>> You're not really introducing a successful Kanban here, you're just
>> clarifying that there should be a set number of workstations.
>>
>> Our current system is like a gigantic open space with hundreds of
>> half-finished pieces, and a dozen workers keep on going from one to
>> another with no strong pattern. The proposed system is to limit the
>> number of half-finished pieces fighting for the workers attention at any
>> given time, by setting a clear number of workstations.
>>
>> A true Kanban would be an interface between developers and reviewers,
>> where reviewers define what type of change they have to review to
>> complete production objectives, *and* developers would strive to produce
>> enough to keep the kanban above the red line, but not too much (which
>> would be piling up waste).
>>
>> Without that discipline, Kanbans are useless. Unless the developers
>> adapt what they work on based on release objectives, you don't really
>> reduce waste/inventory at all, it just piles up waiting for available
>> "runway slots". As I said in my original email, the main issue here is
>> the imbalance between too many people proposing changes and not enough
>> people caring about the project itself enough to be trusted with core
>> reviewers rights.
>>
>> This proposal is not solving that, so it is not the miracle cure that
>> will end all developers frustration, nor is it turning our push-based
>> model into a sane pull-based one. The only way to be truly pull-based is
>> to define a set of production objectives and have those objectives
>> trickle up to the developers so that they don't work on something else.
>> The solution is about setting release cycle goals and strongly
>> communicating that everything out of those goals is clearly priority 2.
>>
>> Now I'm not saying this is a bad idea. Having too many reviews to
>> consider at the same time dilutes review attention to the point where we
>> don't finalize anything. Having runway slots makes sure there is a focus
>> on a limited set of features at a time, which increases the chances that
>> those get finalized.
>>
> 
> I found this response to be very insightful, thank you.
> 
> I feel like this idea is essentially trying to figure out how to apply
> an agile process to nova.  Lots and lots of people have tried to figure
> out how to make it work for open source, and there's several reasons
> that it just doesn't.  This came up in a thread last year here:
> 
> http://lists.openstack.org/pipermail/openstack-dev/2013-April/007872.html/
> 
> With that said, I really do appreciate the hunger to find new and better
> ways to manage our work.  It's certainly needed and I hope to
> continuously improve.
> 
> It seems one of the biggest benefits of this sort of proposal is rate
> limiting how often we say "yes" so that we have more confidence that we
> can follow up on things we say "yes" to.  That is indeed an improvement.
>  We made a pass at trying some similar before, where a blueprint
> required nova-core sponsors to get approved.  It was a total flop as a
> very small subset of nova-core participated, so the process was
> abandoned.  I'm curious what makes this significantly different.  If the
> people doing the review aren't the ones that approve something to get a
> slot, then I'm not sure how we would have any idea it's actually going
> to get reviewed.  Maybe it's close enough to the same thing, and we just
> need to be more aggressive with the process to get people to participate
> in it?

I think the difference is slot selection would just be Nova drivers. I
think there is an assumption in the old system that everyone in Nova
core wants to prioritize the blueprints. I think there are a bunch of
folks in Nova core that are happy having signaling from Nova drivers on
high priority things to review. (I know I'm in that camp.)

Lacking that we all have picking algorithms to hack away at the 500 open
reviews. Which basically means it's a giant random queue.

Having a few blueprints that *everyone* is looking at also has the
advantage that the context for the bits in question will tend to be
loaded into multiple people's heads at the same time, so is something
that's discussable.

Will it fix the issue, not sure, but it's an idea.

	-Sean

-- 
Sean Dague
http://dague.net