Open Stack

Excerpts from Doug Hellmann's message of 2017-11-10 13:11:45 -0500:
> Excerpts from Clint Byrum's message of 2017-11-08 23:15:15 -0800:
> > Excerpts from Samuel Cassiba's message of 2017-11-08 08:27:12 -0800:
> > > On Tue, Nov 7, 2017 at 3:28 PM, Erik McCormick
> > > <emccormick at cirrusseven.com> wrote:
> > > > Hello Ops folks,
> > > >
> > > > This morning at the Sydney Summit we had a very well attended and very
> > > > productive session about how to go about keeping a selection of past
> > > > releases available and maintained for a longer period of time (LTS).
> > > >
> > > > There was agreement in the room that this could be accomplished by
> > > > moving the responsibility for those releases from the Stable Branch
> > > > team down to those who are already creating and testing patches for
> > > > old releases: The distros, deployers, and operators.
> > > >
> > > > The concept, in general, is to create a new set of cores from these
> > > > groups, and use 3rd party CI to validate patches. There are lots of
> > > > details to be worked out yet, but our amazing UC (User Committee) will
> > > > be begin working out the details.
> > > >
> > > > Please take a look at the Etherpad from the session if you'd like to
> > > > see the details. More importantly, if you would like to contribute to
> > > > this effort, please add your name to the list starting on line 133.
> > > >
> > > > https://etherpad.openstack.org/p/SYD-forum-upstream-lts-releases
> > > >
> > > > Thanks to everyone who participated!
> > > >
> > > > Cheers,
> > > > Erik
> > > >
> > > > __________________________________________________________________________
> > > > OpenStack Development Mailing List (not for usage questions)
> > > > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > > 
> > > In advance, pardon the defensive tone. I was not in a position to
> > > attend, or even be in Sydney. However, as this comes across the ML, I
> > > can't help but get the impression this effort would be forcing more
> > > work on already stretched teams, ie. deployment-focused development
> > > teams already under a crunch as contributor count continues to decline
> > > in favor of other projects inside and out of OpenStack.
> > > 
> > 
> > I suspect if LTS's become a normal part of OpenStack, most deployment
> > projects will decline to support the interim releases. We can infer this
> > from the way Ubuntu is used. This might actually be a good thing for the
> > chef OpenStack community. 3 out of 3.5 of you can focus on the LTS bits,
> > and the 0.5 person can do some best effort to cover the weird corner
> > case of "previous stable release to master".
> > 
> > The biggest challenge will be ensuring that the skip-level upgrades
> > work. The current grenade based upgrade jobs are already quite a bear to
> > keep working IIRC. I've not seen if chef or any of the deployment projects
> > test upgrades like that.
> > 
> > However, if people can stop caring much about the interim releases and
> > just keep "previous LTS to master" upgrades working, then that might be
> > good for casual adoption.
> > 
> > Personally I'd rather we make it easier to run "rolling release"
> > OpenStack. Maybe we can do that if we stop cutting stable releases every
> > 6 months.
> > 
> 
> We should stop calling what we're talking about "LTS". It isn't
> going to match the expectations of anyone receiving LTS releases
> for other products, at least at first. Perhaps "Deployer Supported"
> or "User Supported" are better terms for what we're talking about.
> 

I believe this state we're in is a stop-gap on the way to the full
LTS. People are already getting stuck. We're going to help them stay stuck
by upstreaming bug fixes.  We should be mindful of that and provide a way
to get less-stuck. The LTS model from other projects has proven quite
popular, and it would make sense for us to embrace it if our operators
are hurting with the current model, which I believe they are.

> In the "LTS" room we did not agree to stop cutting stable releases
> or to start supporting upgrades directly from N-2 (or older) to N.
> Both of those changes would require modifying the support the
> existing contributor base has committed to provide.
> 

Thanks, I am just inferring those things from what was agreed on. However,
It would make a lot of sense to discuss the plans for the future, even
if we don't have data from the present proposal.

> Fast-forward upgrades will still need to run the migration steps
> of each release in order, one at a time. The team working on that
> is going to produce a document describing what works today so we
> can analyze it for ways to improve the upgrade experience, for both
> fast-forward and "regular" upgrades.  That was all discussed in a
> separate session.
> 

We are what we test. If we're going to test fast-forwards, how far into
the past do we test? It makes a lot of sense to pick a release every few
cycles and say "we're going to support fast-forwards from this point
to master for the next 2 years". Then users can safely clamber their
way onto that release, and spend the next 2 years paying down debt,
knowing they have a chance for a skip-level upgrade. It's not a model
I recommend, but I do think it's the way many companies work today.

Anyway, let me be clear. I think we should listen to our operators. I
wasn't at the forum, so I couldn't do that there. But what I have seen
at every org I've been at, is one of two things:

1) Running close to master, merging in and deploying on a regular
   cadence. Confidence in the pipeline and actual CD nirvana. This
   is rare, and requires commitment and trust at multiple levels. But
   it's the way I personally recommend people run OpenStack.

2) Falling 1 release behind the moment they deploy the stable release,
   and then falling further and further behind from there.

This is my own anecdotal evidence, but we have some data to back it up
too. As of the April survey[1] we see that _most_ users were on Mitaka or
earlier. Mitaka went EOL in _April_. So users are getting run over by the
EOL policy, and no amount of "you should upgrade faster" is going to fix
that. Perhaps we should slow down the tagging. Some have suggested going
to once per year. That makes some sense to me. Especially if we continue
to support CD'ing from master.

Then we can ask the operators if they would prefer that we stop EOL'ing
things out from under them. We can make it a community goal to have a
"feature" that is "you can upgrade from the last one" and have "the last
one" be something older than 6 months, maybe even older than 1 year.

[1] https://www.openstack.org/assets/survey/April2017SurveyReport.pdf