Open Stack

On Wed, Feb 25, 2015 at 10:31:58AM +0100, Thierry Carrez wrote:
> Robert Collins wrote:
> >> It's also worth noting that we were on a 3-month cycle at the start of
> >> OpenStack. That was dropped after a cataclysmic release that managed the
> >> feat of (a) not having anything significant done, and (b) have out of
> >> date documentation and translations.
> > 
> > Oh! https://wiki.openstack.org/w/index.php?title=Obsolete:3MonthReleaseCycle&direction=prev&oldid=14016
> > appears to be the page about that, but my google-fu isn't turning up a
> > post-mortem of the C release which prompted the change. Perhaps some
> > old-hand like you can fill us in on the details :).
> 
> After the Austin release (a code drop, really), we were on a three-month
> time-based cycle with no intermediary milestone. The first cycle (Bexar)
> worked relatively well, the second one (Cactus) not so well. There were
> multiple reasons for that:
> 
> - Software was growing up in Bexar and we introduced more testing, but
> we were unable to detect and fix all the release blockers in time for
> the release date, resulting in a broken Bexar release. To address that
> in Cactus, we increased the duration of the frozen period, but then that
> didn't leave enough time for development, hence nothing getting done.

I wasn't involved in OpenStack back then, but is it not true that
OpenStack has changed almost beyond recognition since those early
days. The number of projects has certainly increased massively, as
has the number of contributors, both individual & corporate. There
is a hell of alot of testing now and people doing continuous deployment
of trunk, so I find it hard to believe that trunk is so unstable that
we can't do a release at any time we choose, nor that we have to have
such long freezes that we don't do get time for dev.

> - Not starting a release cycle with a global F2F gathering (no
> intermediary Design Summit) actually proved quite disastrous: no reset
> of community, no alignment on goal, no discussion on what to work on.
> That resulted in a limbo period after the Bexar release. That is why I'm
> so insistent on aligning our development cycles with our Design Summits.
> I've seen where we go when we don't.

I don't know about other projects so much, but I don't really see the
design summit as a positive thing wrt planning the next release. For
a start the design summit is atbout 5 weeks after the trunk opens for
development, so if people wait for the summit do do planning they have
thrown away half of the first milestones' development time. It is also
not inclusive as a decision making forum because we cannot assume every
one is able to make it to the summit in person, and even if they are
present, people often can't get involved in all sessions they wish to
due to conflicting schedules. If release planning were done via email
primarily, with IRC for cases where realtime planning is needed, it
would be more effective IMHO.  IOW i think the summit would be better
off if were explicitly not associated with releases, but rather be a
general forum for collaboration were we talk through ideas or do code
sprints, and more.

> > [...]
> >> I may appear defensive on my answers, but it's not my goal to defend the
> >> current system: it's just that most of those proposals generally ignore
> >> the diversity of the needs of the teams that make OpenStack possible, to
> >> focus on a particular set of contributors' woes. I'm trying to bring
> >> that wider perspective in -- the current system is a trade-off and the
> >> result of years of evolution, not an arbitrary historic choice that we
> >> can just change at will.
> > 
> > I agree that its not arbitrary and that changing it requires some
> > appropriate wide-spread consultation; OTOH the benefits of rolling and
> > higher frequency releases are really substantial: but we have to
> > backup the change with the appropriate engineering (such as reducing
> > our frictions that cause teams practicing CD to be weeks behind tip)
> > to make it feasible. My greatest concern about the proposals happening
> > now is that we may bite off more than we can chew. OTGH the reality is
> > that all the negative things multiply out, so we probably need to just
> > start somewhere and /do/ to give us the space to fix other things.
> 
> As I said, I'd very much like us to "release" a lot more often. I just
> don't see how we can do it without:
> 
> - abandoning the idea of translating the software
> - admit that docs will always lag code
> - dropping stable branch maintenance (and only fix vulnerabilities in
> master)

I don't really buy that. If we have 6 month cycles with 1 month freeze
for docs & i18n, vs 2 months cycles with 2 weeks freeze for docs & i18n
the latter is a win in terms of dev time vs stablization time, giving
docs & i18n 50% more time in aggregate.

> That would all make my life (and the life of most feature-focused
> developers) a *lot* easier. I'm not convinced that's the best solution
> for our downstream users, though. I think they like docs,
> translations(?), stable branches and backported security fixes.
> 
> PS: as a data point, at the last summit, in the stable branch session,
> there was a whole group of downstream users crashing the session to ask
> for releasing *less* often ("we can't keep up"). I think that's crazy
> too, but that shows 6 months is really a trade-off.

I'd be wary of taking that those users literally. IME working on RHEL
everyone wants very infrequent unchanging releases, except for their
own favourite new features to be added. So you end up with infrequent
releases but with major feature backports to those so called "stable"
branches. This diverts massive amounts of developer resource away from
doing net new upstream work, while at the same time destablizing the
stable branches. I already see this happening with OpenStack due to the
severe time delay in getting access to features over the 6 month dev
cycle.  BTW, when I talk about stable branches in this context, I'm
more referring to the vendor's OpenStack products, rather than upstream
stable branches. Upstream stable branches dont accept any features at
all, which leaves users who want plain upstream OpenStack stuck waiting
for 6 months. This actually pushes users towards vendors who are willing
to provide & support the features they want on a more reasonable time
scale. This is great for us vendors but IMHO is really a reflection on
the upstream release model

By doing more frequent releases you would reduce the pressure to backport
features to stable branches, and reduce pressure to jam in lots of major
features at the last minute in master which risks destablizing trunk too
which is bad.

I think the key take away is that you must not force users to upgrade to
new releases in an unreasonable timeframe - you need to give them the
ability to have a reasonably long deployment life when they pick a release,
and allow them to skip releases if they want to upgrade. Since we only
official supporting N-1 to N upgrades, we are effectively forcing people
to try to deploy every 6 months. If they want to skip a cycle and stay
deloyed for 12 months, then we're causing them pain by saying they have
to upgrade to this intermediate release first, before going to the one
they actually want to get to.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|