Open Stack

Hi,

On piątek, 5 listopada 2021 18:47:13 CET Sean Mooney wrote:
> On Fri, 2021-11-05 at 11:53 -0400, Mohammed Naser wrote:
> > On Fri, Nov 5, 2021 at 10:39 AM Thierry Carrez <thierry at openstack.org> 
wrote:
> > > Hi everyone,
> > > 
> > > The (long) document below reflects the current position of the release
> > > management team on a popular question: should the OpenStack release
> > > cadence be changed? Please note that we only address the release
> > > management / stable branch management facet of the problem. There are
> > > other dimensions to take into account (governance, feature deprecation,
> > > supported distros...) to get a complete view of the debate.
> > > 
> > > Introduction
> > > ------------
> > > 
> > > The subject of how often OpenStack should be released has been regularly
> > > debated in the OpenStack community. OpenStack started with a 3-month
> > > release cycle, then switched to 6-month release cycle starting with
> > > Diablo. It is often thought of a release management decision, but it is
> > > actually a much larger topic: a release cadence is a trade-off between
> > > pressure to release more often and pressure to release less often,
> > > coming in from a lot of different stakeholders. In OpenStack, it is
> > > ultimately a Technical Committee decision. But that decision is informed
> > > by the position of a number of stakeholders. This document gives
> > > historical context and describes the current release management team
> > > position.
> > > 
> > > The current trade-off
> > > ---------------------
> > > 
> > > The main pressure to release more often is to make features available to
> > > users faster. Developers get a faster feedback loop, hardware vendors
> > > ensure software is compatible with their latest products, and users get
> > > exciting new features. "Release early, release often" is a best practice
> > > in our industry -- we should generally aim at releasing as often as
> > > possible.
> > > 
> > > But that is counterbalanced by pressure to release less often. From a
> > > development perspective, each release cycle comes with some process
> > > overhead. On the integrators side, a new release means packaging and
> > > validation work. On the users side, it means pressure to upgrade. To
> > > justify that cost, there needs to be enough user-visible benefit (like
> > > new features) in a given release.
> > > 
> > > For the last 10 years for OpenStack, that balance has been around six
> > > months. Six months let us accumulate enough new development that it was
> > > worth upgrading to / integrating the new version, while giving enough
> > > time to actually do the work. It also aligned well with Foundation
> > > events cadence, allowing to synchronize in-person developer meetings
> > > date with start of cycles.
> > > 
> > > What changed
> > > ------------
> > > 
> > > The major recent change affecting this trade-off is that the pace of new
> > > development in OpenStack slowed down. The rhythm of changes was divided
> > > by 3 between 2015 and 2021, reflecting that OpenStack is now a mature
> > > and stable solution, where accessing the latest features is no longer a
> > > major driver. That reduces some of the pressure for releasing more
> > > often. At the same time, we have more users every day, with larger and
> > > larger deployments, and keeping those clusters constantly up to date is
> > > an operational challenge. That increases the pressure to release less
> > > often. In essence, OpenStack is becoming much more like a LTS
> > > distribution than a web browser -- something users like moving slow.
> > > 
> > > Over the past years, project teams also increasingly decoupled
> > > individual components from the "coordinated release". More and more
> > > components opted for an independent or intermediary-released model,
> > > where they can put out releases in the middle of a cycle, making new
> > > features available to their users. This increasingly opens up the
> > > possibility of a longer "coordinated release" which would still allow
> > > development teams to follow "release early, release often" best
> > > practices. All that recent evolution means it is (again) time to
> > > reconsider if the 6-month cadence is what serves our community best, and
> > > in particular if a longer release cadence would not suit us better.
> > > 
> > > The release management team position on the debate
> > > --------------------------------------------------
> > > 
> > > While releasing less often would definitely reduce the load on the
> > > release management team, most of the team work being automated, we do
> > > not think it should be a major factor in motivating the decision. We
> > > should not adjust the cadence too often though, as there is a one-time
> > > cost in switching our processes. In terms of impact, we expect that a
> > > switch to a longer cycle will encourage more project teams to adopt a
> > > "with-intermediary" release model (rather than the traditional "with-rc"
> > > single release per cycle), which may lead to abandoning the latter,
> > > hence simplifying our processes. Longer cycles might also discourage
> > > people to commit to PTL or release liaison work. We'd probably need to
> > > manage expectations there, and encourage more frequent switches (or
> > > create alternate models).
> > > 
> > > If the decision is made to switch to a longer cycle, the release
> > > management team recommends to switch to one year directly. That would
> > > avoid changing it again anytime soon, and synchronizing on a calendar
> > > year is much simpler to follow and communicate. We also recommend
> > > announcing the change well in advance. We currently have an opportunity
> > > of making the switch when we reach the end of the release naming
> > > alphabet, which would also greatly simplify the communications around
> > > the change.
> > > 
> > > Finally, it is worth mentioning the impact on the stable branch work.
> > > Releasing less often would likely impact the number of stable branches
> > > that we keep on maintaining, so that we do not go too much in the past
> > > (and hit unmaintained distributions or long-gone dependencies). We
> > > currently maintain releases for 18 months before they switch to extended
> > > maintenance, which results in between 3 and 4 releases being maintained
> > > at the same time. We'd recommend switching to maintaining one-year
> > > releases for 24 months, which would result in between 2 and 3 releases
> > > being maintained at the same time. Such a change would lead to longer
> > > maintenance for our users while reducing backporting work for our
> > > developers.
> > 
> > Thanks for the write up Thierry.
> > 
> > I wonder what are the thoughts of the community of having LTS + normal
> > releases so that we can have the power of both?  I guess that is
> > essentially what we have with EM, but I guess we could introduce a way to
> > ensure that operators can just upgrade LTS to LTS.
> 
> if we were to intoduce LTS release we would have to agree on what they were 
as
> a compunity and we would need to support roling upgrade between LTS versions
> 
> that would reqruie all distibuted project like nova to ensur that lts to lts
> rpc and db compatitblity is maintained instead of the current N+1 
guarunetees
> we have to day.

Not only that but also cross project communication, like e.g. nova <-> neutron 
needs to work fine between such LTS releases.

> 
> i know that would make some downstream happy as perhaps we could align our 
FFU
> support with THE LTS cadance but i would hold my breath on that.
> 
> as a developer i woudl presonally prefer to have shorter cycle upstream with
> uprades supporte aross a more then n+1 e.g. release every 2 months but keep
> rolling upgrade compatiablty for at least 12 months or someting like that.
> the release with intermeiday lifecyle can enable that while still allowign
> use  to have a longer or shorter planing horizon depending on the project 
and
> its veliocity.
> 
> > It can complicate things a bit from a CI and project management side,
> > but I think it
> > could solve the problem for both sides that need want new features +
> > those who want
> > stability?
> 
> it might but i suspect that it will still not align with distros
> canonical have a new lts every 2 years and redhat has a new release every
> 18~months or so based on every 3rd release
> 
> the lts idea i think has merrit but we likely would have to maintain at 
least
> 2 lts release in paralel to make it work.
> 
> so something like  1 lts release a year maintained for 2 years with normal
> release every 6 months that are only maintianed for 6 months each project
> woudl keep rolling upgrade compatitblity ideally between lts release rather
> then n+1 as a new mimium. the implication of this is that we would want to
> have grenade jobs testing latest lts to master upgrade compatiblity in
> additon to n to n+1 where those differ.
> > > --
> > > Thierry Carrez (ttx)
> > > On behalf of the OpenStack Release Management team


-- 
Slawek Kaplonski
Principal Software Engineer
Red Hat
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20211107/35d6a7d2/attachment.sig>