Open Stack

Excerpts from Clint Byrum's message of 2017-11-11 08:41:15 -0800:
> Excerpts from Doug Hellmann's message of 2017-11-10 13:11:45 -0500:
> > Excerpts from Clint Byrum's message of 2017-11-08 23:15:15 -0800:
> > > Excerpts from Samuel Cassiba's message of 2017-11-08 08:27:12 -0800:
> > > > On Tue, Nov 7, 2017 at 3:28 PM, Erik McCormick
> > > > <emccormick at cirrusseven.com> wrote:
> > > > > Hello Ops folks,
> > > > >
> > > > > This morning at the Sydney Summit we had a very well attended and very
> > > > > productive session about how to go about keeping a selection of past
> > > > > releases available and maintained for a longer period of time (LTS).
> > > > >
> > > > > There was agreement in the room that this could be accomplished by
> > > > > moving the responsibility for those releases from the Stable Branch
> > > > > team down to those who are already creating and testing patches for
> > > > > old releases: The distros, deployers, and operators.
> > > > >
> > > > > The concept, in general, is to create a new set of cores from these
> > > > > groups, and use 3rd party CI to validate patches. There are lots of
> > > > > details to be worked out yet, but our amazing UC (User Committee) will
> > > > > be begin working out the details.
> > > > >
> > > > > Please take a look at the Etherpad from the session if you'd like to
> > > > > see the details. More importantly, if you would like to contribute to
> > > > > this effort, please add your name to the list starting on line 133.
> > > > >
> > > > > https://etherpad.openstack.org/p/SYD-forum-upstream-lts-releases
> > > > >
> > > > > Thanks to everyone who participated!
> > > > >
> > > > > Cheers,
> > > > > Erik
> > > > >
> > > > > __________________________________________________________________________
> > > > > OpenStack Development Mailing List (not for usage questions)
> > > > > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > > > 
> > > > In advance, pardon the defensive tone. I was not in a position to
> > > > attend, or even be in Sydney. However, as this comes across the ML, I
> > > > can't help but get the impression this effort would be forcing more
> > > > work on already stretched teams, ie. deployment-focused development
> > > > teams already under a crunch as contributor count continues to decline
> > > > in favor of other projects inside and out of OpenStack.
> > > > 
> > > 
> > > I suspect if LTS's become a normal part of OpenStack, most deployment
> > > projects will decline to support the interim releases. We can infer this
> > > from the way Ubuntu is used. This might actually be a good thing for the
> > > chef OpenStack community. 3 out of 3.5 of you can focus on the LTS bits,
> > > and the 0.5 person can do some best effort to cover the weird corner
> > > case of "previous stable release to master".
> > > 
> > > The biggest challenge will be ensuring that the skip-level upgrades
> > > work. The current grenade based upgrade jobs are already quite a bear to
> > > keep working IIRC. I've not seen if chef or any of the deployment projects
> > > test upgrades like that.
> > > 
> > > However, if people can stop caring much about the interim releases and
> > > just keep "previous LTS to master" upgrades working, then that might be
> > > good for casual adoption.
> > > 
> > > Personally I'd rather we make it easier to run "rolling release"
> > > OpenStack. Maybe we can do that if we stop cutting stable releases every
> > > 6 months.
> > > 
> > 
> > We should stop calling what we're talking about "LTS". It isn't
> > going to match the expectations of anyone receiving LTS releases
> > for other products, at least at first. Perhaps "Deployer Supported"
> > or "User Supported" are better terms for what we're talking about.
> > 
> 
> I believe this state we're in is a stop-gap on the way to the full
> LTS. People are already getting stuck. We're going to help them stay stuck
> by upstreaming bug fixes.  We should be mindful of that and provide a way
> to get less-stuck. The LTS model from other projects has proven quite
> popular, and it would make sense for us to embrace it if our operators
> are hurting with the current model, which I believe they are.
> 
> > In the "LTS" room we did not agree to stop cutting stable releases
> > or to start supporting upgrades directly from N-2 (or older) to N.
> > Both of those changes would require modifying the support the
> > existing contributor base has committed to provide.
> > 
> 
> Thanks, I am just inferring those things from what was agreed on. However,
> It would make a lot of sense to discuss the plans for the future, even
> if we don't have data from the present proposal.
> 
> > Fast-forward upgrades will still need to run the migration steps
> > of each release in order, one at a time. The team working on that
> > is going to produce a document describing what works today so we
> > can analyze it for ways to improve the upgrade experience, for both
> > fast-forward and "regular" upgrades.  That was all discussed in a
> > separate session.
> > 
> 
> We are what we test. If we're going to test fast-forwards, how far into
> the past do we test? It makes a lot of sense to pick a release every few
> cycles and say "we're going to support fast-forwards from this point
> to master for the next 2 years". Then users can safely clamber their
> way onto that release, and spend the next 2 years paying down debt,
> knowing they have a chance for a skip-level upgrade. It's not a model
> I recommend, but I do think it's the way many companies work today.

That seems like a good approach. When the team working on fast-forward
upgrades finishes the process document it should become more clear how
we could potentially start running upgrade tests, and what changes might
be needed to support doing so. Having some tags for supporting
fast-forward and skip-level upgrades would be good, too, so deployers
can understand easily what sort of upgrade models are supported by
different projects.

> Anyway, let me be clear. I think we should listen to our operators. I
> wasn't at the forum, so I couldn't do that there. But what I have seen
> at every org I've been at, is one of two things:
> 
> 1) Running close to master, merging in and deploying on a regular
>    cadence. Confidence in the pipeline and actual CD nirvana. This
>    is rare, and requires commitment and trust at multiple levels. But
>    it's the way I personally recommend people run OpenStack.
> 
> 2) Falling 1 release behind the moment they deploy the stable release,
>    and then falling further and further behind from there.
> 
> This is my own anecdotal evidence, but we have some data to back it up
> too. As of the April survey[1] we see that _most_ users were on Mitaka or
> earlier. Mitaka went EOL in _April_. So users are getting run over by the
> EOL policy, and no amount of "you should upgrade faster" is going to fix
> that. Perhaps we should slow down the tagging. Some have suggested going
> to once per year. That makes some sense to me. Especially if we continue
> to support CD'ing from master.

The problem with stable maintenance isn't so much the number of
releases involved. Stable branches become more brittle as they age,
because the underlying OS changes with upgrades in dependencies.
We discussed a bunch of different ways to address that, from
containers to static images to managing a "frozen" repo of packages
to install during tests. Any decisions on what approach to use will
be left to the people actually doing the work, along with the folks
maintaining the tools used (in particular, the infra and QA teams).

In preparation for the summit I went back through all of the notes
I could find about stable branches from previous summits. The first
mention of stable branches I found was at the Folsom summit, and
that was a discussion of doing stable releases "more often," which
implies that we had the stable branches at least as early as Essex.
I wasn't around before Folsom, so I'm not sure when they actually
started.  The first mention of an LTS release I found was the Juno
summit, which was later than I expected.

No one in the room disputed the assertion that what we're doing for
stable releases is insufficient.  We are all trying to listen to
users' needs.  Continuing to just say "we should do LTS releases"
however doesn't acknowledge the other long standing fact, which is
that over all of the time we have talked about it we have had *no
contributors willing to actually support an LTS release model
upstream.*

We act like people have been saying "no, you are not allowed to
maintain branches for longer than the stable team says is OK," or
"no, we'll never provide an LTS release," but that's not how our
community works. The policies are set by the contributors. If there
are no contributors for an LTS release, there will be no LTS release.
If there *are* contributors, then we'll find a way to make some
sort of LTS model work within the other constraints we have.

It seems now that we have people saying they would do some amount
of maintenance work (probably less than we try to do for stable
branches under our current model), if they could. The first change,
what has been proposed, is to give them a place to do the work.
Then we'll see if anyone actually does it, and if so we can plan
further improvements.

> Then we can ask the operators if they would prefer that we stop EOL'ing
> things out from under them. We can make it a community goal to have a
> "feature" that is "you can upgrade from the last one" and have "the last
> one" be something older than 6 months, maybe even older than 1 year.
> 
> [1] https://www.openstack.org/assets/survey/April2017SurveyReport.pdf
>