Open Stack

Robert Collins wrote:
>> It's also worth noting that we were on a 3-month cycle at the start of
>> OpenStack. That was dropped after a cataclysmic release that managed the
>> feat of (a) not having anything significant done, and (b) have out of
>> date documentation and translations.
> 
> Oh! https://wiki.openstack.org/w/index.php?title=Obsolete:3MonthReleaseCycle&direction=prev&oldid=14016
> appears to be the page about that, but my google-fu isn't turning up a
> post-mortem of the C release which prompted the change. Perhaps some
> old-hand like you can fill us in on the details :).

After the Austin release (a code drop, really), we were on a three-month
time-based cycle with no intermediary milestone. The first cycle (Bexar)
worked relatively well, the second one (Cactus) not so well. There were
multiple reasons for that:

- Software was growing up in Bexar and we introduced more testing, but
we were unable to detect and fix all the release blockers in time for
the release date, resulting in a broken Bexar release. To address that
in Cactus, we increased the duration of the frozen period, but then that
didn't leave enough time for development, hence nothing getting done.

- Not starting a release cycle with a global F2F gathering (no
intermediary Design Summit) actually proved quite disastrous: no reset
of community, no alignment on goal, no discussion on what to work on.
That resulted in a limbo period after the Bexar release. That is why I'm
so insistent on aligning our development cycles with our Design Summits.
I've seen where we go when we don't.

The post-mortem (at the Diablo summit) recognized our inability as a
group (including QA, docs, stable branch, vulnerability management) to
do final release work more often than every 6 months. Milestones were
introduced so that we can communicate features faster to our users
(those wanting fresh stuff could consume milestones), and then focus QA
/ docs / stable support on the final release only.

Now, the funny thing is... this solution is very similar to what is
proposed in this thread. Intermediary releases every two months to get
code out really fast, and final releases which trigger stable
maintenance. So what happened ?

What happened then is what will happen if we reboot the exact same
solution now: developers ignoring milestones (or intermediary releases)
to focus only on the "real", stable-supported one. And now milestones
are just dates in a calendar.

So when I say that won't solve anything and devs will keep on focusing
on the "real" release, it's not just a gut feeling. We've been there. I
like to learn from history and not repeat the same mistakes.

> [...]
>> I may appear defensive on my answers, but it's not my goal to defend the
>> current system: it's just that most of those proposals generally ignore
>> the diversity of the needs of the teams that make OpenStack possible, to
>> focus on a particular set of contributors' woes. I'm trying to bring
>> that wider perspective in -- the current system is a trade-off and the
>> result of years of evolution, not an arbitrary historic choice that we
>> can just change at will.
> 
> I agree that its not arbitrary and that changing it requires some
> appropriate wide-spread consultation; OTOH the benefits of rolling and
> higher frequency releases are really substantial: but we have to
> backup the change with the appropriate engineering (such as reducing
> our frictions that cause teams practicing CD to be weeks behind tip)
> to make it feasible. My greatest concern about the proposals happening
> now is that we may bite off more than we can chew. OTGH the reality is
> that all the negative things multiply out, so we probably need to just
> start somewhere and /do/ to give us the space to fix other things.

As I said, I'd very much like us to "release" a lot more often. I just
don't see how we can do it without:

- abandoning the idea of translating the software
- admit that docs will always lag code
- dropping stable branch maintenance (and only fix vulnerabilities in
master)

That would all make my life (and the life of most feature-focused
developers) a *lot* easier. I'm not convinced that's the best solution
for our downstream users, though. I think they like docs,
translations(?), stable branches and backported security fixes.

PS: as a data point, at the last summit, in the stable branch session,
there was a whole group of downstream users crashing the session to ask
for releasing *less* often ("we can't keep up"). I think that's crazy
too, but that shows 6 months is really a trade-off.

-- 
Thierry Carrez (ttx)