Open Stack

On Tue, 2015-02-24 at 12:05 +0100, Thierry Carrez wrote:
> Daniel P. Berrange wrote:
> > [...]
> > The key observations
> > ====================
> > 
> > The first key observation from the schedule is that although we have
> > a 6 month release cycle, we in fact make 4 releases in that six
> > months because there are 3 milestones releases approx 6-7 weeks apart
> > from each other, in addition to the final release. So one of the key
> > burdens of a more frequent release cycle is already being felt, to
> > some degree.
> > 
> > The second observation is that thanks to the need to support a
> > continuous deployment models, the GIT master branches are generally
> > considered to be production ready at all times. The tree does not
> > typically go through periods of major instability that can be seen
> > in other projects, particular those which lack such comprehensive
> > testing infrastructure.
> > 
> > The third observation is that due to the relatively long cycle, and
> > increasing amounts of process, the work accomplished during the
> > cycles is becoming increasingly bursty. This is in turn causing
> > unacceptably long delays for contributors when their work is unlucky
> > enough to not get accepted during certain critical short windows of
> > opportunity in the cycle.
> > 
> > The first two observations strongly suggest that the choice of 6
> > months as a cycle length is a fairly arbitrary decision that can be
> > changed without unreasonable pain. The third observation suggests a
> > much shorter cycle length would smooth out the bumps and lead to a
> > more efficient & satisfying development process for all involved.
> 
> I think you're judging the cycle from the perspective of developers
> only. 6 months was not an arbitrary decision. Translations and
> documentation teams basically need a month of feature/string freeze in
> order to complete their work. Since we can't reasonably freeze one month
> every 2 months, we picked 6 months.

Actually, this is possible: look at Linux, it freezes for 10 weeks of a
12 month release cycle (or 6 weeks of an 8 week one).  More on this
below.

> It's also worth noting that we were on a 3-month cycle at the start of
> OpenStack. That was dropped after a cataclysmic release that managed the
> feat of (a) not having anything significant done, and (b) have out of
> date documentation and translations.
> 
> While I agree that the packagers and stable teams can opt to skip a
> release, the docs, translations or security teams don't really have that
> luxury... Please go beyond the developers needs and consider the needs
> of the other teams.
> 
> Random other comments below:
> 
> > [...]
> > Release schedule
> > ----------------
> > 
> > First the releases would probably be best attached to a set of
> > pre-determined fixed dates that don't ever vary from year to year.
> > eg releses happen Feb 1st, Apr 1st, Jun 1st, Aug 1st, Oct 1st, and
> > Dec 1st. If a particular release slips, don't alter following release
> > dates, just shorten the length of the dev cycle, so it becomes fully
> > self-correcting. The even numbered months are suggested to avoid a
> > release landing in xmas/new year :-)
> 
> The Feb 1 release would probably be pretty empty :)
> 
> > [...]
> > Stable branches
> > ---------------
> > 
> > The consequences of a 2 month release cycle appear fairly severe for
> > the stable branch maint teams at first sight. This is not, however,
> > an insurmountable problem. The linux kernel shows an easy way forward
> > with their approach of only maintaining stable branches for a subset
> > of major releases, based around user / vendor demand. So it is still
> > entirely conceivable that the stable team only provide stable branch
> > releases for 2 out of the 6 yearly releases. ie no additional burden
> > over what they face today. Of course they might decide they want to
> > do more stable branches, but maintain each for a shorter time. So I
> > could equally see them choosing todo 3 or 4 stable branches a year.
> > Whatever is most effective for those involved and those consuming
> > them is fine.
> 
> Stable branches may have the luxury of skipping releases and designate a
> "stable" one from time to time (I reject the Linux comparison because
> the kernel is at a very different moment in software lifecycle). The
> trick being, making one release "special" is sure to recreate the peak
> issues you're trying to solve.

I don't disagree with the observation about different points in the
lifecycle, but perhaps it might be instructive to ask if the linux
kernel ever had a period in its development history that looks somewhat
like OpenStack does now.  I would claim it did: before 2.6, we had the
odd/even develop/stabilise cycle.  The theory driving it was that we
needed a time for everyone to develop then a time for everyone to help
make stable.  You yourself said this in the other thread:

> Joe Gordon wrote:
> > [...]
> > I think a lot of the frustration with our current cadence comes out of
> > the big stop everything (development, planning etc.), and stabilize the
> > release process. Which in turn isn't really making things more stable.
> 
> I guess I have a different position. I think the release candidate
> period is the only thing that makes your code drops actually usable.
> It's the only moment in the cycle where integrators test. It's the
> only
> moment in the cycle where developers work on bugs they did not file
> themselves, but focus on a project-wide priority list of release
> blockers. If you remove that period, then nobody will ever work on
> release blockers that do not directly affect them. Shorten that period
> to one week, and no integrator will have the time to run a proper QA
> program to detect those release blockers.

This is a precise echo of the driver for the Linux kernel odd/even
develop/stabilise cycle.  However, the develop/stabilise cycle lead to a
huge amount of frustration from people who wanted to add new features
while we were stabilising (and from distributions who needed a release
while we were developing).

In the beginning, for Linux Kernel: 0.9/1.0; 1.1/1.2; 1.3/2.0; 2.1/2.2
all had relatively short cycles for develop/stabilise, but the
frustrations in the system meant that by the time we got to 2.3/2.4 the
cycle was out of control and 2.5 exploded for us and led to the adoption
of the new model for 2.6 that you see today.

The ultimate way we resolved the tensions was to acknowledge that
developers will develop.  If they're interested, they'll help you
stabilise, but if not they'll just back bite about their frustrations
trying to get their next feature in.  Conversely, you have a whole
ecosystem (the OpenStack distributions) interested in stability, so
they'll help you stabilise. So the Linux Kernel now do an overlapping
develop/release cycle, where it looks like we stabilise for n-2 weeks of
the n week cycle, but in reality developers develop the N+1 release in
that time-frame and the people interested in stability stabilise the
release.  The result is that a lot of the developer frustration
evaporated (they only get yelled at if they try to discuss patches with
maintainers in the 2-week merge window period) and stability hasn't
really suffered.

This is a long winded way of saying that Daniel has the right idea (in
my opinion) but it would depend on adopting different processes than
today.

James