[openstack-dev] [all] The future of the integrated release

Thierry Carrez thierry at openstack.org
Thu Aug 7 08:49:50 UTC 2014

Stefano Maffulli wrote:
> On Wed 06 Aug 2014 02:10:23 PM PDT, Michael Still wrote:
>>  - we rate limit the total number of blueprints under code review at
>> any one time to a fixed number of "slots". I secretly prefer the term
>> "runway", so I am going to use that for the rest of this email. A
>> suggested initial number of runways was proposed at ten.
> oooooh, I like the 'slots/runaway model'. Sounds to me like kan ban (in 
> the Toyota sense not the hipster developer sense).
> A light in my head just went on.
> Let me translate what you're thinking about in other terms: the 
> slot/runaway model would switch what is now a push model into a "pull" 
> model. Currently we have patches coming in, pushed up for review. We 
> have then on gerrit reviewers and core reviewers shuffling through 
> these changesets, doing work and approve/comment. The reviewers have 
> little to no way to notice when they're overloaded and managers have no 
> way either. There is no way to identify when the process is suffering, 
> slowing down or not satisfying demand, if not when the backlog blows 
> up. As recent discussions demonstrate, this model is failing under our 
> growth.
> By switching to a model where we have a set of slots/runaway (buckets, 
> in Toyota's terminology) reviewers would have a clear way to *pull* new 
> reviews into their workstations to be processed. It's as simple as a 
> supermaket aisle: when there is no more pasta on the shelf, a clerk 
> goes in the backend and gets more pasta to refurbish the shelf. There 
> is no sophisticated algorithm to predict demand: it's the demand of 
> pasta that drives new pull requests (of pasta or changes to review).
> This pull mechanism would help make it very visible where the 
> bottlenecks are. At Toyota, for example, the amount of kanbans is the 
> visible way to understand the capacity of the plant. The amount of 
> slots/runaways would probably give us similar overview of the capacity 
> of each project and give us tools to solve bottlenecks before they 
> become emergencies.

As an ex factory IT manager, I feel compelled to comment on that :)
You're not really introducing a successful Kanban here, you're just
clarifying that there should be a set number of workstations.

Our current system is like a gigantic open space with hundreds of
half-finished pieces, and a dozen workers keep on going from one to
another with no strong pattern. The proposed system is to limit the
number of half-finished pieces fighting for the workers attention at any
given time, by setting a clear number of workstations.

A true Kanban would be an interface between developers and reviewers,
where reviewers define what type of change they have to review to
complete production objectives, *and* developers would strive to produce
enough to keep the kanban above the red line, but not too much (which
would be piling up waste).

Without that discipline, Kanbans are useless. Unless the developers
adapt what they work on based on release objectives, you don't really
reduce waste/inventory at all, it just piles up waiting for available
"runway slots". As I said in my original email, the main issue here is
the imbalance between too many people proposing changes and not enough
people caring about the project itself enough to be trusted with core
reviewers rights.

This proposal is not solving that, so it is not the miracle cure that
will end all developers frustration, nor is it turning our push-based
model into a sane pull-based one. The only way to be truly pull-based is
to define a set of production objectives and have those objectives
trickle up to the developers so that they don't work on something else.
The solution is about setting release cycle goals and strongly
communicating that everything out of those goals is clearly priority 2.

Now I'm not saying this is a bad idea. Having too many reviews to
consider at the same time dilutes review attention to the point where we
don't finalize anything. Having runway slots makes sure there is a focus
on a limited set of features at a time, which increases the chances that
those get finalized.

Thierry Carrez (ttx)

More information about the OpenStack-dev mailing list