[openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time

Sean Dague sean at dague.net
Fri May 5 12:12:29 UTC 2017

On 05/05/2017 07:16 AM, Bogdan Dobrelya wrote:
> On 05.05.2017 2:09, Zane Bitter wrote:
>> On 04/05/17 10:14, Thierry Carrez wrote:
>>> We started with no stable branches, we were just producing releases and
>>> ensuring that updates vaguely worked from N-1 to N. There were a lot of
>>> distributions, and they all maintained their own stable branches,
>>> handling backport of critical fixes. That is a pretty classic upstream /
>>> downstream model.
>>> Some of us (including me) spotted the obvious duplication of effort
>>> there, and encouraged distributions to share that stable branch
>>> maintenance work rather than duplicate it. Here the stable branches were
>>> born, mostly through a collaboration between Red Hat developers and
>>> Canonical developers. All was well. Nobody was saying LTS back then
>>> because OpenStack was barely usable so nobody wanted to stay on any
>>> given version for too long.
>> Heh, if you go back _that_ far then upgrades between versions basically
>> weren't feasible, so everybody stayed on a given version for too long.
>> It's true that nobody *wanted* to though :D
> I may be wrong, but I have a strong perception that even when major
> upgrades run smooth and flawless, operators tend to operate legacy
> enterprise software for long, long, long. It would be an utopia to
> expect any changes here, IMO. Unless the major shift comes to
> enterprises for the very software delivery paradigm (infrastructure as a
> code, blue/green deployments done by CD pipeline promoting stable
> changes from trunk commits, unicorns all around), which is an utopia
> even more though.
>>> Maintaining stable branches has a cost. Keeping the infrastructure that
>>> ensures that stable branches are actually working is a complex endeavor
>>> that requires people to constantly pay attention. As time passed, we saw
>>> the involvement of distro packagers become more limited. We therefore
>>> limited the number of stable branches (and the length of time we
>>> maintained them) to match the staffing of that team.
>> I wonder if this is one that needs revisiting. There was certainly a
>> time when closing a branch came with a strong sense of relief that you
>> could stop nursing the gate. I personally haven't felt that way in a
> This. What we really need, IMO, that sense of relief, but inversed,
> which is *operators* of enterprises world to feel it and start nursing
> their 3rd party CI gates *upstream* once a stable branch created! And
> hopefully never closed, for the time they do care of it. It sounds fair
> to open source software. As Jay noted, one shall get that one had put,
> which is a direct function of how much do one care and gives away.
> So perhaps there is a (naive?) option #3: Do not support or nurse gates
> for stable branches upstream. Instead, only create and close them and
> attach 3rd party gating, if asked by contributors willing to support and
> nurse their gates.

I think it's important to clarify the amount of infrastructure that goes
into testing OpenStack. We build a whole cloud, from source, installing
~ 200 python packages, many from git trees, configure and boot 100+ VMs
on it in different ways. And do that with a number of different default

Nothing prevents anyone from building a kilo branch in a public github
and doing their own CI against it. But we've never seen anyone do that,
because there is a lot of work in maintaining a CI system. A lot of
expertise needed to debug when things go wrong. Anyone can captain a
ship when it's a sunny day. Only under failures do we see what is
required to actually keep the ship afloat.

You could always drop all the hard parts of CI, actually testing the
trees build a full running cloud. But at that point, it becomes very odd
to call it a stable branch, as it is far less rigorous in validation
than master.

At any rate, this basically comes up every year, and I don't think the
fundamental equation has changed.


Sean Dague

More information about the OpenStack-dev mailing list