Open Stack

On Fri, 2021-11-05 at 11:53 -0400, Mohammed Naser wrote:
> On Fri, Nov 5, 2021 at 10:39 AM Thierry Carrez <thierry at openstack.org> wrote:
> > 
> > Hi everyone,
> > 
> > The (long) document below reflects the current position of the release
> > management team on a popular question: should the OpenStack release
> > cadence be changed? Please note that we only address the release
> > management / stable branch management facet of the problem. There are
> > other dimensions to take into account (governance, feature deprecation,
> > supported distros...) to get a complete view of the debate.
> > 
> > Introduction
> > ------------
> > 
> > The subject of how often OpenStack should be released has been regularly
> > debated in the OpenStack community. OpenStack started with a 3-month
> > release cycle, then switched to 6-month release cycle starting with
> > Diablo. It is often thought of a release management decision, but it is
> > actually a much larger topic: a release cadence is a trade-off between
> > pressure to release more often and pressure to release less often,
> > coming in from a lot of different stakeholders. In OpenStack, it is
> > ultimately a Technical Committee decision. But that decision is informed
> > by the position of a number of stakeholders. This document gives
> > historical context and describes the current release management team
> > position.
> > 
> > The current trade-off
> > ---------------------
> > 
> > The main pressure to release more often is to make features available to
> > users faster. Developers get a faster feedback loop, hardware vendors
> > ensure software is compatible with their latest products, and users get
> > exciting new features. "Release early, release often" is a best practice
> > in our industry -- we should generally aim at releasing as often as
> > possible.
> > 
> > But that is counterbalanced by pressure to release less often. From a
> > development perspective, each release cycle comes with some process
> > overhead. On the integrators side, a new release means packaging and
> > validation work. On the users side, it means pressure to upgrade. To
> > justify that cost, there needs to be enough user-visible benefit (like
> > new features) in a given release.
> > 
> > For the last 10 years for OpenStack, that balance has been around six
> > months. Six months let us accumulate enough new development that it was
> > worth upgrading to / integrating the new version, while giving enough
> > time to actually do the work. It also aligned well with Foundation
> > events cadence, allowing to synchronize in-person developer meetings
> > date with start of cycles.
> > 
> > What changed
> > ------------
> > 
> > The major recent change affecting this trade-off is that the pace of new
> > development in OpenStack slowed down. The rhythm of changes was divided
> > by 3 between 2015 and 2021, reflecting that OpenStack is now a mature
> > and stable solution, where accessing the latest features is no longer a
> > major driver. That reduces some of the pressure for releasing more
> > often. At the same time, we have more users every day, with larger and
> > larger deployments, and keeping those clusters constantly up to date is
> > an operational challenge. That increases the pressure to release less
> > often. In essence, OpenStack is becoming much more like a LTS
> > distribution than a web browser -- something users like moving slow.
> > 
> > Over the past years, project teams also increasingly decoupled
> > individual components from the "coordinated release". More and more
> > components opted for an independent or intermediary-released model,
> > where they can put out releases in the middle of a cycle, making new
> > features available to their users. This increasingly opens up the
> > possibility of a longer "coordinated release" which would still allow
> > development teams to follow "release early, release often" best
> > practices. All that recent evolution means it is (again) time to
> > reconsider if the 6-month cadence is what serves our community best, and
> > in particular if a longer release cadence would not suit us better.
> > 
> > The release management team position on the debate
> > --------------------------------------------------
> > 
> > While releasing less often would definitely reduce the load on the
> > release management team, most of the team work being automated, we do
> > not think it should be a major factor in motivating the decision. We
> > should not adjust the cadence too often though, as there is a one-time
> > cost in switching our processes. In terms of impact, we expect that a
> > switch to a longer cycle will encourage more project teams to adopt a
> > "with-intermediary" release model (rather than the traditional "with-rc"
> > single release per cycle), which may lead to abandoning the latter,
> > hence simplifying our processes. Longer cycles might also discourage
> > people to commit to PTL or release liaison work. We'd probably need to
> > manage expectations there, and encourage more frequent switches (or
> > create alternate models).
> > 
> > If the decision is made to switch to a longer cycle, the release
> > management team recommends to switch to one year directly. That would
> > avoid changing it again anytime soon, and synchronizing on a calendar
> > year is much simpler to follow and communicate. We also recommend
> > announcing the change well in advance. We currently have an opportunity
> > of making the switch when we reach the end of the release naming
> > alphabet, which would also greatly simplify the communications around
> > the change.
> > 
> > Finally, it is worth mentioning the impact on the stable branch work.
> > Releasing less often would likely impact the number of stable branches
> > that we keep on maintaining, so that we do not go too much in the past
> > (and hit unmaintained distributions or long-gone dependencies). We
> > currently maintain releases for 18 months before they switch to extended
> > maintenance, which results in between 3 and 4 releases being maintained
> > at the same time. We'd recommend switching to maintaining one-year
> > releases for 24 months, which would result in between 2 and 3 releases
> > being maintained at the same time. Such a change would lead to longer
> > maintenance for our users while reducing backporting work for our
> > developers.
> > 
> 
> Thanks for the write up Thierry.
> 
> I wonder what are the thoughts of the community of having LTS + normal releases
> so that we can have the power of both?  I guess that is essentially what we have
> with EM, but I guess we could introduce a way to ensure that operators can just
> upgrade LTS to LTS.
if we were to intoduce LTS release we would have to agree on what they were as a compunity and
we would need to support roling upgrade between LTS versions

that would reqruie all distibuted project like nova to ensur that lts to lts rpc and db compatitblity is
maintained instead of the current N+1 guarunetees we have to day.

i know that would make some downstream happy as perhaps we could align our FFU support with THE LTS cadance but i
would hold my breath on that.

as a developer i woudl presonally prefer to have shorter cycle upstream with uprades supporte aross a more then n+1
e.g. release every 2 months but keep rolling upgrade compatiablty for at least 12 months or someting like that.
the release with intermeiday lifecyle can enable that while still allowign use  to have a longer or shorter planing horizon depending
on the project and its veliocity.

> 
> It can complicate things a bit from a CI and project management side,
> but I think it
> could solve the problem for both sides that need want new features +
> those who want
> stability?

it might but i suspect that it will still not align with distros
canonical have a new lts every 2 years and redhat has a new release every 18~months or so based on every 3rd release

the lts idea i think has merrit but we likely would have to maintain at least 2 lts release in paralel to make it work.

so something like  1 lts release a year maintained for 2 years with normal release every 6 months that are only maintianed for 6 months 
each project woudl keep rolling upgrade compatitblity ideally between lts release rather then n+1 as a new mimium.
the implication of this is that we would want to have grenade jobs testing latest lts to master upgrade compatiblity in additon to n to n+1 where those differ.


> 
> > --
> > Thierry Carrez (ttx)
> > On behalf of the OpenStack Release Management team
> > 
> 
>