Open Stack

Joe Gordon wrote:
> [...]
> I think a lot of the frustration with our current cadence comes out of
> the big stop everything (development, planning etc.), and stabilize the
> release process. Which in turn isn't really making things more stable.

I guess I have a different position. I think the release candidate
period is the only thing that makes your code drops actually usable.
It's the only moment in the cycle where integrators test. It's the only
moment in the cycle where developers work on bugs they did not file
themselves, but focus on a project-wide priority list of release
blockers. If you remove that period, then nobody will ever work on
release blockers that do not directly affect them. Shorten that period
to one week, and no integrator will have the time to run a proper QA
program to detect those release blockers.

I understand that from your developer perspective it's annoying to have
to work on project-wide priorities rather than your own, and therefore
you'd like to get rid of those -- but the resulting drop in quality is
just something we can't afford.

> So I propose we keep the 6 month release cycle, but change the
> development cycle from a 6 month one with 3 intermediate milestones to a
> 6 week one with a milestone at the end of it.
> 
> What this actually means:
> 
>   * Stop approving blueprints for specific stable releases, instead just
>     approve them and target them to milestones.
>       o Milestones stop becoming Kilo-1, Kilo-2, Kilo-3 etc. and just
>         become 1,2,3,4,5,6,7,8,9 etc.
>       o If something misses what was previously known as Kilo-3 it has
>         to wait a week for what for milestone 4.
>   * Development focuses on milestones only. So 6 week cycle with say 1
>     week of stabilization, finish things up before each milestone
>   * Process of cutting a stable branch (planning, making the branch,
>     doing release candidates, testing etc.) is done by a dedicated
>     stable branch team. And it should be done based on a specific milestone
>   * Goal: Change the default development planning mode to ignore stable
>     branches, and allow for us to think of things in terms of the number
>     of milestones needed, not will it make the stable branch or not

I don't think that would solve any of the issues you mentioned:
> Current issues
>   * 3 weeks of feature freeze for all projects at the end of each cycle
>     (3 out of the 26 feature blocked)

So you'll have 3 x 1 week of feature freeze for all projects, instead of
1 x 3 weeks. That will be less efficient (integrators need a >1week
feature freeze period to actually start testing a non-moving target),
more painful (have to organize it 3 times instead of 1), and likely
inefficient (takes generally more than one week to find critical bugs,
develop the fix, and get it reviewed). And in the end, it's still 3 out
of the 26 feature blocked.

>   * 3 weeks of release candidates. Once a candidate is cut development
>     is open for next release. While this is good in theory, not much
>     work actually starts on next release. 

That is not really what I observe. People start landing their feature in
the master branch starting the day after RC1. I actually observe the
opposite: too many people switching to master development, and not
enough people working on RC2+ bugs.

>   * some projects have non priority feature freezes and at Milestone 2
>     (so 9 out of 26 weeks restricted in those projects)

That was their own choice. I for one was really surprised that they
would freeze earlier -- I think 3 weeks is the right balance.

>   * vicious development circle
>       o vicious circle:
>           + big push right to land lots of features right before the release

I think you'll have exactly the same push before the "stable" milestone
(or the one that will be adopted by $DISTRO).

>>           + a lot of effort is spent getting the release ready
>>           + after the release people are a little burnt out and take it
>>             easy until the next summit

Not convinced the burn out will be less significant with 4 releases
instead of one every 6 months. Arguably it will be more distributed. But
the main reason for the drop in activity between release and summit is
that a lot of companies organize their team gatherings around that time
(to prepare for summit).

>>           + Blueprints have to be re-discussed and re-approved for the
>>             next cycle
>>           + makes it hard to land blueprints early in the cycle casing
>>             the bug rush at the end of the cycle, see step 1

Re-approving is each project choice. Personally I think revetting
already-approved specs is a bit of madness.

>>       o Makes it hard to plan things across a release

I don't see how the proposed change really addresses that. Do you plan
to have 4 midcycle sprints ?

>>       o This actually destabilizes things right as we go into the
>>         stabilization period (We actually have great data on this too)
>>       o Means postponing blueprints that miss a deadline several months

I think transforming 3 milestones + 1 release into 3 intermediary
releases and a "stable" final release will exhibit *exactly* the same
issues.

-- 
Thierry Carrez (ttx)