Open Stack

On Tue, Feb 24, 2015 at 12:05:17PM +0100, Thierry Carrez wrote:
> Daniel P. Berrange wrote:
> > [...]
> > The key observations
> > ====================
> > 
> > The first key observation from the schedule is that although we have
> > a 6 month release cycle, we in fact make 4 releases in that six
> > months because there are 3 milestones releases approx 6-7 weeks apart
> > from each other, in addition to the final release. So one of the key
> > burdens of a more frequent release cycle is already being felt, to
> > some degree.
> > 
> > The second observation is that thanks to the need to support a
> > continuous deployment models, the GIT master branches are generally
> > considered to be production ready at all times. The tree does not
> > typically go through periods of major instability that can be seen
> > in other projects, particular those which lack such comprehensive
> > testing infrastructure.
> > 
> > The third observation is that due to the relatively long cycle, and
> > increasing amounts of process, the work accomplished during the
> > cycles is becoming increasingly bursty. This is in turn causing
> > unacceptably long delays for contributors when their work is unlucky
> > enough to not get accepted during certain critical short windows of
> > opportunity in the cycle.
> > 
> > The first two observations strongly suggest that the choice of 6
> > months as a cycle length is a fairly arbitrary decision that can be
> > changed without unreasonable pain. The third observation suggests a
> > much shorter cycle length would smooth out the bumps and lead to a
> > more efficient & satisfying development process for all involved.
> 
> I think you're judging the cycle from the perspective of developers
> only. 6 months was not an arbitrary decision. Translations and
> documentation teams basically need a month of feature/string freeze in
> order to complete their work. Since we can't reasonably freeze one month
> every 2 months, we picked 6 months.

I'm actually trying to judge it from the POV of users, not just
developers. I find it pretty untenable that in the fast moving
world of cloud, users have to wait as long as 6 months for a
feature to get into a openstack release, often much longer.

I'm not familiar with how the translations works, but if they are
waiting until the freeze before starting translation work I'd
say that is a mistaken approach. Obviously during active dev part
of the cycle, some translated strings are in flux, so if translation
was taking place in parallel there could be some wasted effort, but
I'd expect that to be the minority case. I think the majority of
translation work can be done in parallel with dev work and the freeze
time just needs to tie up the small remaining bits.

With a 2 month cycle, I could see at least 2 weeks of freeze time
being available, so that'd be 6 weeks in total across 3 releases.
Bearing in mind there'd be 1/3 of the volume of changes in 2 months,
this could actually be a net win on aggregate. So I don't think this
is a totally intractable problem for translation work.

Documentation is again something I'd expect to be done more or less
in parallel with dev. The same point about the number of changes made
in a 2 month cycle, vs the extra freeze time on aggregate applies.
So again I don't think it is intractable either.

> It's also worth noting that we were on a 3-month cycle at the start of
> OpenStack. That was dropped after a cataclysmic release that managed the
> feat of (a) not having anything significant done, and (b) have out of
> date documentation and translations.

I don't think (a) is really relevant at all. New releases do not have to
imply significant new features. It is perfectly reasonable if some of the
releases happen to have had the bulk of their work focus on bug fixing or
stabalization, instead of headline features. We shouldn't be afraid of
this happening & should indeed celebrate if we had a release with few
features and lots of bugs fixed.

In my own experiance releasing libvirt on a strictly once a month schedule,
we do sometimes have releases with not very many features included, but lots
of bugs & I think it has worked out pretty much fine. Also with the volume of
features being submitted to the core openstack projects, I don't think there
is much risk these days of being light on features.

> > [...]
> > Release schedule
> > ----------------
> > 
> > First the releases would probably be best attached to a set of
> > pre-determined fixed dates that don't ever vary from year to year.
> > eg releses happen Feb 1st, Apr 1st, Jun 1st, Aug 1st, Oct 1st, and
> > Dec 1st. If a particular release slips, don't alter following release
> > dates, just shorten the length of the dev cycle, so it becomes fully
> > self-correcting. The even numbered months are suggested to avoid a
> > release landing in xmas/new year :-)
> 
> The Feb 1 release would probably be pretty empty :)

I don't think that's really a problem worth worrying about. It might have
25% less features, but that'll make it a more stable release. In any case
not all parts of the world take the same weeks off for xmas/new year, so
there's still plenty of people working during this time in many regions.

> > [...]
> > Stable branches
> > ---------------
> > 
> > The consequences of a 2 month release cycle appear fairly severe for
> > the stable branch maint teams at first sight. This is not, however,
> > an insurmountable problem. The linux kernel shows an easy way forward
> > with their approach of only maintaining stable branches for a subset
> > of major releases, based around user / vendor demand. So it is still
> > entirely conceivable that the stable team only provide stable branch
> > releases for 2 out of the 6 yearly releases. ie no additional burden
> > over what they face today. Of course they might decide they want to
> > do more stable branches, but maintain each for a shorter time. So I
> > could equally see them choosing todo 3 or 4 stable branches a year.
> > Whatever is most effective for those involved and those consuming
> > them is fine.
> 
> Stable branches may have the luxury of skipping releases and designate a
> "stable" one from time to time (I reject the Linux comparison because
> the kernel is at a very different moment in software lifecycle). The
> trick being, making one release "special" is sure to recreate the peak
> issues you're trying to solve.
> 
> I'm actually more concerned about the teams that don't have the luxury
> of skipping a release (like the vulnerability management team). Docs and
> translations, as mentioned above, will also have a hard time producing
> quality docs and translations every 2 months with a very short freeze
> period.

It would be reasonable for the vulnerability team to take the decision
that they'll support fixes for master, and any branches that the stable
team decide to support. ie they would not neccessarily have to commit
to shipping fixes for every single release made. Or they could do a
progressive policy, where critical issues go to every releases, but
moderate issues only go to releases with stable branches. Even if they
did decide to support each release on an equal footing that's not quite
as bad as it seems. The more frequent the releases are, the less code
churn there will have been, and so you increase the chances that the
same fix will cherry-pick to the previous release with no conflicts.
So having x3 the number of releases, doesn't imply x3 the amount of
work. It might be only x2 or less extra work.

> > [...]
> > Upgrades & deprecation
> > ----------------------
> > 
> > It is common right now for projects to say upgrades are only
> > supported between releases N-1 and N. ie to go from Icehouse
> > to Kilo, you need to first deploy Juno. This is passable when
> > you're talking 6 month gaps between cycles, but when there are
> > 2 month gaps it is not reasonable to expect everyone to be
> > moving fast enough to keep up with every release. If an
> > organization's beurocracy means they can't deploy more often
> > than every 12 months, forcing them to deploy the 5 intermediate
> > releases to run upgrade scripts is quite unpleasant. We would
> > likely have to look at officially allowing upgrades between
> > any (N-3, N-2, N-1) to N. From a database POV this should not
> > be hard, since the DB migration scripts don't have any built
> > in assumptions about this. Similarly the versioned objects used
> > by Nova are quite flexible in this regard too, as long as the
> > compat code isn't deleted too soon.
> > 
> > Deprecation warnings would need similar consideration. It would
> > not be sufficient to deprecate in one release and delete in the
> > next. We'd likely want to say that depecations last for a given
> > time period rather than number of releases, eg 6 months. This
> > could be easily handled by simply including the date of initial
> > deprecation in the deprecation message. It would thus be clear
> > when the feature will be removed to all involved.
> 
> The QA team was complaining about having to support more than 2 branches
> at a time. Not sure testing a full matrix of upgrades will be a happier
> thought.
> 
> I may appear defensive on my answers, but it's not my goal to defend the
> current system: it's just that most of those proposals generally ignore
> the diversity of the needs of the teams that make OpenStack possible, to
> focus on a particular set of contributors' woes. I'm trying to bring
> that wider perspective in -- the current system is a trade-off and the
> result of years of evolution, not an arbitrary historic choice that we
> can just change at will.

I really not trying to focus on the developers woes. I'm trying to focus on
making OpenStack better serve our users. My main motiviation here is that I
think we're doing a pretty terrible job at getting work done that is important
to our users in a timely manner. This is caused by a workflow & release cycle
that is negatively impacting the developers. I think we can fix this without
putting an unreasonable burden on other teams, but I accept some teams may
need to do more work. If this is so, then I don't think this is a blocker,
it is just a sign that the project needs to focus on providing more resources
to the teams impacted in that way.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|