Open Stack

I was writing this mail for the past few days, but the nova thread
today prompted me to finish it off & send it :-)

At the Paris summit I started thinking about the effect that the
release cycle has on the operation of the project. In the months
since I've come to believe that this 6 month cycle could be the
cause of a number of the problems the project is facing. This mail
will attempt to illustrate how those problems manifest themselves,
and how they might be addressed by shortening the release cycle
to just 2 months. NB, I don't think the release cycle is the only
problem[1] that needs addressing, there is no single silver bullet,
but it is an important one.

If you don't fancy reading the whole mail just now skip down to
the heading "The modest proposal" for the core points that matter.

The Kilo Schedule
=================

Currently we have a 6 month release cycle with three milestone
releases before GA time. Looking at the current cycle dates [3]

  Sep 30, 2014: Kilo development opens (wk 40)
  Oct 16, 2014: Juno Release           (wk 42)
  Nov  6, 2014: Paris Design Summit    (wk 45)
  Dec 18, 2014: Kilo-1 milestone       (wk 51)
  Feb  5, 2015: Kilo-2 milestone       (wk  6)
  Mar 19, 2015: Kilo-3 milestone       (wk 12)
  Apr 30, 2015: Kilo Release           (wk 18)

For the Nova team, there were some more specific schedule points.

 - Oct  7, 2014: Kilo specs open for review [4]  (wk 41)
 - Dec 18, 2014: Spec approval soft deadline     (wk 51)
 - Jan 22, 2015: Spec approval hard deadline     (wk  4)
 - Feb  5, 2015: Nonpriority soft feature freeze (wk  6)
 - Feb 20, 2015: Nonpriority hard feature freeze (wk  8)
 - Mar 19, 2015: Priority feature freeze         (wk 12)

Other teams had their own specific schedule points to.

Calculating the durations of "interesting" phases

 - Approx 10 week window for proposing specs
 - Approx  5 week extra for spec exception requests
 - Approx 16 week window for merging features
 - Approx  2 week extra for freeze exception requests
 - Approx  6 week extra for priority features

The 10 week window for specs is a little misleading though as the
list of so called "project priorites" for Nova was not actually
decided until the design summit. So the main things being approved
in the first 5 weeks were those that were denied feature freeze
exception from Juno. New proposals were primarily evaluated in
the second 5 week window, plus the possible spec exception extension
if requested by the submitter.

Similarly the 16 week window for merging features is also somewhat
misleading. Alot of the spec proposals were not reviewed until
shortly before the spec approval soft deadline. So a spec approved
on Dec 18 may have had as little as 7 weeks to get the code merged
before feature freeze. Since the Kilo dev cycle crossed the year
end, many contributors will have lost another 2 weeks of potential
dev & review time.


Effects of the schedule deadlines
=================================

Those who are familiar with OpenStack development for a while, will
understand the pain that the deadlines can bring, so take preemptive
action to maximise their chances of success. For example, while code
cannot be merged with the spec approved, it is possible to start
writing code & submitting it for review before the spec is approved.
This is not without its own risk, for example, review comments on
the specs may entail rewrite of existing code, or their spec may not
get approved for Kilo at all, forcing a wait till Liberty. On balance
though, such "risk start" of development is likely worthwhile given
the difficulty in getting attention to code reviews at any time.

Life is harder for people who are new to OpenStack development, or
those who come up with an idea for OpenStack at the wrong point
during the schedule. For example, someone joining the OpenStack
community for the first time and submitting a proposal for a bit
of work on Jan 2nd has a pretty low chance of getting their spec
approved for Kilo given the point in the dev cycle. It will be
not uncommon for them to be told to resubmit for Liberty.

Nova is a bit more organized this time around, so Liberty specs
can be submitted already, but none the less, assuming they are
successful in getting their feature merged in Liberty, it is not
going to be available to end users until approx Oct 15th. eg a
9 month gap between idea & availability. If they are unsuccessful
at getting reviews, through no fault of their own, they can find
their work bumped for another release. This would add another 6
months before their feature makes it out to end users deploying
clouds.

A gap of 9 months is a very long time to wait for inclusion of a
feature in the cloud world. A gap of 15 months is even more severe.
This is a not uncommon occurrance though, particularly for new
contributors.

The consequences of missing the current release cycle are such that
there are inevitably a very large number of spec and code freeze
exception requests made. Core team members will often get their
inboxes filled with mail from contributors essentially begging for
support for the exception. There is also often a big rush of code
submitted for review close to the deadlines, in an attempt to avoid
getting pushed to the next cycle.

This results in a very bursty development cycle with big peaks of
activity which put significant pressure on core teams ability to
process the workload. Different members of the core teams have
different view points on how to deal with this. Some will take the
view that as much as possible should be approved in order to minimise
the number of people who have to suffer the significant time delay
to the next release. Others take the view that the focus should be
on a small number of more strategic things and bug fixes. Both are
valid views with differing pros & cons, but neither are entirely
satisfactory and will often result in unhappiness amongst both the
team and contributors.

The key observation here is that when the release cycle is 6 months
long, saying no to a contributor imposes a very significant time
penalty. There are a few possible consequences, some of which are

 - The contributor resubmits the idea for the next cycle and does
   nothing else in current dev cycle

 - The contributor resubmits the idea for the next cycle and works
   on something else in current dev cycle

 - The contributor abandons the idea completely, and does nothing
   else in the current dev cycle

 - The contributor abandons the idea completely, and works on
   something else during the dev cycle

 - The contributor works on it anyway and ships a non-upstream
   fork of openstack to their users

 - The contributor abandons OpenStack entirely and goes to work on
   different project with higher chance of success

Most of these consequences are quite harmful to the success of
OpenStack, especially the last three. Supposedly the Nova project
priorities are there to encourage people whose features are rejected,
to instead work on a priority item. There is little evidence that
this is actually happening though. Some of the priorites have had
almost no work done, while others are worked on merely by people
who were already involved in them.


The key observations
====================

The first key observation from the schedule is that although we have
a 6 month release cycle, we in fact make 4 releases in that six
months because there are 3 milestones releases approx 6-7 weeks apart
from each other, in addition to the final release. So one of the key
burdens of a more frequent release cycle is already being felt, to
some degree.

The second observation is that thanks to the need to support a
continuous deployment models, the GIT master branches are generally
considered to be production ready at all times. The tree does not
typically go through periods of major instability that can be seen
in other projects, particular those which lack such comprehensive
testing infrastructure.

The third observation is that due to the relatively long cycle, and
increasing amounts of process, the work accomplished during the
cycles is becoming increasingly bursty. This is in turn causing
unacceptably long delays for contributors when their work is unlucky
enough to not get accepted during certain critical short windows of
opportunity in the cycle.

The first two observations strongly suggest that the choice of 6
months as a cycle length is a fairly arbitrary decision that can be
changed without unreasonable pain. The third observation suggests a
much shorter cycle length would smooth out the bumps and lead to a
more efficient & satisfying development process for all involved.


The modest proposal
===================

Based on these observations I would suggest that OpenStack aim to
switch to a development cycle that is exactly 2 months long.
ie do 6 releases a year.

This will have a number of consequences which I'd expect to be
beneficial to the project on balance.

Release schedule
----------------

First the releases would probably be best attached to a set of
pre-determined fixed dates that don't ever vary from year to year.
eg releses happen Feb 1st, Apr 1st, Jun 1st, Aug 1st, Oct 1st, and
Dec 1st. If a particular release slips, don't alter following release
dates, just shorten the length of the dev cycle, so it becomes fully
self-correcting. The even numbered months are suggested to avoid a
release landing in xmas/new year :-)

Specs approvals
---------------

With such a short cycle, it would no longer make sense for specs and
blueprints to be submitted against a particular release. Instead
specs would be freely submitted and reviewed at *any* time throughout
the year. Spec approval would focus exclusively on ensuring the
design was sensible and within scope of the project for the forseeable
future. So approval would say nothing about what release the code
must be merged in. Specs approval would instead notionally be valid
for a fixed amount of time, perhaps 12 months from time of approval.
After that time the spec could be re-evaluated to ensure it still
made conceptual sense, but this should be a lightweight process with
little burden for reviewers or submitters alike.

Code approvals
--------------

During each 2 month release cycle, anyone with an approved spec is
free to submit code for review to implement the spec in question. It
would not be neccessary to fully implement a spec in 1 cycle, some
simple specs might take 1 cycle, some complex specs merge code over
the course of 3, 4, 5 or more cycles. Either way is fine, as long as
the spec still has valid approval. During a 2 month release cycle,
a project may decide to prioritize the review of code for certain
specs to meet key project objectives. Such prioritization though
is clearly separate from actual spec approval. At some point in the
2 month window, there would be a feature freeze deadline, probably
somewhere between 4 and 6 weeks into the cycle depending on how much
time is considered neccessary for pre-release stabalization.

The key benefit of approving spec designs separately from priortizing
& scheduling the merging of code, is that we are always providing a
carrot to potential contributors to continue to be part of OpenStack.
Once their spec is approved, it is clear that their proposal is
desired by the project. We are not telling them to go away for
another 6 months and submit the design again with no guarantee of
acceptance next time around either.

In the 2 month cycles, there will inevitably still be more code
submitted than there is capacity to review & merge. So there will
still be decisions to be made about which reviews to prioritize
and exceptions that need granting or rejecting. Likewise my previous
raised issues about the need to address the core reviewer bottleneck
still stand and need addressing in parallel. The difference is that
the time between the exception being rejected and the next merge
window opening will be a mere 2-3 weeks, as opposed to 2-3 months.
This will again serve to maintain the contributors interest in
participating with openstack. It will also likely reduce the work
they have to do to continue rebasing their code before it has a
chance of merging again. It also means that when a feature is
rejected in one dev cycle, their users are not forced to wait 9+
months for it to possibly appear in a future release. There will
be a reasonable chance of simple features getting merged and released
to users in as little as 2 months.

Stable branches
---------------

The consequences of a 2 month release cycle appear fairly severe for
the stable branch maint teams at first sight. This is not, however,
an insurmountable problem. The linux kernel shows an easy way forward
with their approach of only maintaining stable branches for a subset
of major releases, based around user / vendor demand. So it is still
entirely conceivable that the stable team only provide stable branch
releases for 2 out of the 6 yearly releases. ie no additional burden
over what they face today. Of course they might decide they want to
do more stable branches, but maintain each for a shorter time. So I
could equally see them choosing todo 3 or 4 stable branches a year.
Whatever is most effective for those involved and those consuming
them is fine.

Downstream vendors
------------------

Downstream product vendors would face a similar decision about how
they wished to produce. Some OS vendors may choose to ship all stable
releases in their distros as they become available, others may choose
to focus on a subset of the releases. In some cases more frequent
release may actually simplify life for the vendors. For example, if
there is a feature they need which doesn't make the cut for a release
today, they currently have the choice of waiting 6 months for the
next release, or carrying non-upstream patches, or backporting
patches from master. With a 2 month release cycle, it is more
practical for vendors to simply wait 2 months for the next release
to ship. So this would probably reduce the amount of patch
backporting vendors have todo in general, giving them more time to
focus on useful upstream work.

Design summit
-------------

Another consequence of 2 month release cycles would be that the
design summits become entirely detached from the release cycles, as
they'd be on different cycle lengths. So instead of having a focus
on release planning, design summits would likely become a forum for
general ongoing collaboration and discussions. Probably a half &
half merge between the current design summit format and a more
traditional open source conference. This also ought to result in
less pressure for all contributors to attend all summits, and so be
time efficient travelwise and beneficial in cost reduction for
vendors.

More interestingly, is that by detaching them from release cycles,
the summit organizers could have more flexibility with scheduling
the summits. They could be placed on the dates that are most suitable
for desired travel locations, rather than dates suited to release
schedules. Likewise there is more freedom if there was ever a wish
to have design summits and the main track conference separated.

Election process
----------------

I wouldn't suggest any changes to project elections or governance.
It is fine to have elections every 6 months. There is no need for
that to be tied to release cycle length in any way, particularly
as people often serve for more than one release already.

Release naming
--------------

Last but not least, there would be some questions around what this
might mean for release naming. There are many options and I don't
intend to suggest a particular one is best, just highlight some
possibilities that could be considered.

The easiest is to just drop the idea of release names entirely and
have everyone refer to the release versions instead. Another option
is to give each 2 month release its own name, so we'd have to come
up with 3x as many names during the year. The person running the
release naming elections will probably have a heart attack at this
idea :-) Another option is to have names that span a couple of
releases, perhaps all releases for an entire year get a given name.

Upgrades & deprecation
----------------------

It is common right now for projects to say upgrades are only
supported between releases N-1 and N. ie to go from Icehouse
to Kilo, you need to first deploy Juno. This is passable when
you're talking 6 month gaps between cycles, but when there are
2 month gaps it is not reasonable to expect everyone to be
moving fast enough to keep up with every release. If an
organization's beurocracy means they can't deploy more often
than every 12 months, forcing them to deploy the 5 intermediate
releases to run upgrade scripts is quite unpleasant. We would
likely have to look at officially allowing upgrades between
any (N-3, N-2, N-1) to N. From a database POV this should not
be hard, since the DB migration scripts don't have any built
in assumptions about this. Similarly the versioned objects used
by Nova are quite flexible in this regard too, as long as the
compat code isn't deleted too soon.

Deprecation warnings would need similar consideration. It would
not be sufficient to deprecate in one release and delete in the
next. We'd likely want to say that depecations last for a given
time period rather than number of releases, eg 6 months. This
could be easily handled by simply including the date of initial
deprecation in the deprecation message. It would thus be clear
when the feature will be removed to all involved.

Why do releases at all ?
------------------------

Some people might wonder why we should do releases at all. We
already aim to support a continuous deployment model for users
as an option. Continuous deployment, however, is quite a
conceptual jump for many users and organizations in particular.
Their internal software approval processes and bureaucracy can
make this impractical to even consider in the forseeable future.
So if openstack did not do releases, then many downstream vendors
would still end up taking git snapshots and forming periodic
releases around them. The unsatisfactory result would be a
balkanization of OpenStack where each vendor duplicated the others
work with their own release cycles in slightly different ways. To
some extent this happens already as vendors backport features, due
to inability to wait 6 months for next major release. A two month
release cycle though would likely reduce the amount of difference
between what each vendor ships which would be a good thing in
general. Releases also serve as a useful hook for marketing
activities by the project and vendors alike, giving a good
opportunity to promote the new body of features now available to
users. It might be difficult to drum up media enthusiasm to
promote arbitrary git snapshots, but then I'm no marketing
expert.




Regards,
Daniel

[1] http://lists.openstack.org/pipermail/openstack-dev/2014-September/044872.html
[2] http://www.openstack.org/projects/openstack-faq/
[3] https://wiki.openstack.org/wiki/Kilo_Release_Schedule
[4] http://lists.openstack.org/pipermail/openstack-dev/2014-October/047922.html

-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|