Open Stack

 ---- On Fri, 05 Nov 2021 09:26:13 -0500 Thierry Carrez <thierry at openstack.org> wrote ----
 > Hi everyone,
 > 
 > The (long) document below reflects the current position of the release 
 > management team on a popular question: should the OpenStack release 
 > cadence be changed? Please note that we only address the release 
 > management / stable branch management facet of the problem. There are 
 > other dimensions to take into account (governance, feature deprecation, 
 > supported distros...) to get a complete view of the debate.
 > 
 > Introduction
 > ------------
 > 
 > The subject of how often OpenStack should be released has been regularly 
 > debated in the OpenStack community. OpenStack started with a 3-month 
 > release cycle, then switched to 6-month release cycle starting with 
 > Diablo. It is often thought of a release management decision, but it is 
 > actually a much larger topic: a release cadence is a trade-off between 
 > pressure to release more often and pressure to release less often, 
 > coming in from a lot of different stakeholders. In OpenStack, it is 
 > ultimately a Technical Committee decision. But that decision is informed 
 > by the position of a number of stakeholders. This document gives 
 > historical context and describes the current release management team 
 > position.
 > 
 > The current trade-off
 > ---------------------
 > 
 > The main pressure to release more often is to make features available to 
 > users faster. Developers get a faster feedback loop, hardware vendors 
 > ensure software is compatible with their latest products, and users get 
 > exciting new features. "Release early, release often" is a best practice 
 > in our industry -- we should generally aim at releasing as often as 
 > possible.
 > 
 > But that is counterbalanced by pressure to release less often. From a 
 > development perspective, each release cycle comes with some process 
 > overhead. On the integrators side, a new release means packaging and 
 > validation work. On the users side, it means pressure to upgrade. To 
 > justify that cost, there needs to be enough user-visible benefit (like 
 > new features) in a given release.

Thanks Thierry for the detailed write up. 

At the same time, a shorter release which leads to upgrade-often pressure but
it will have fewer number of changes/features, so make the upgrade easy and
longer-release model will have more changes/features that will make upgrade more
complex.


 > 
 > For the last 10 years for OpenStack, that balance has been around six 
 > months. Six months let us accumulate enough new development that it was 
 > worth upgrading to / integrating the new version, while giving enough 
 > time to actually do the work. It also aligned well with Foundation 
 > events cadence, allowing to synchronize in-person developer meetings 
 > date with start of cycles.
 > 
 > What changed
 > ------------
 > 
 > The major recent change affecting this trade-off is that the pace of new 
 > development in OpenStack slowed down. The rhythm of changes was divided 
 > by 3 between 2015 and 2021, reflecting that OpenStack is now a mature 
 > and stable solution, where accessing the latest features is no longer a 
 > major driver. That reduces some of the pressure for releasing more 
 > often. At the same time, we have more users every day, with larger and 
 > larger deployments, and keeping those clusters constantly up to date is 
 > an operational challenge. That increases the pressure to release less 
 > often. In essence, OpenStack is becoming much more like a LTS 
 > distribution than a web browser -- something users like moving slow.
 > 
 > Over the past years, project teams also increasingly decoupled 
 > individual components from the "coordinated release". More and more 
 > components opted for an independent or intermediary-released model, 
 > where they can put out releases in the middle of a cycle, making new 
 > features available to their users. This increasingly opens up the 
 > possibility of a longer "coordinated release" which would still allow 
 > development teams to follow "release early, release often" best 
 > practices. All that recent evolution means it is (again) time to 
 > reconsider if the 6-month cadence is what serves our community best, and 
 > in particular if a longer release cadence would not suit us better.
 > 
 > The release management team position on the debate
 > --------------------------------------------------
 > 
 > While releasing less often would definitely reduce the load on the 
 > release management team, most of the team work being automated, we do 
 > not think it should be a major factor in motivating the decision. We 
 > should not adjust the cadence too often though, as there is a one-time 
 > cost in switching our processes. In terms of impact, we expect that a 
 > switch to a longer cycle will encourage more project teams to adopt a 
 > "with-intermediary" release model (rather than the traditional "with-rc" 
 > single release per cycle), which may lead to abandoning the latter, 
 > hence simplifying our processes. Longer cycles might also discourage 
 > people to commit to PTL or release liaison work. We'd probably need to 
 > manage expectations there, and encourage more frequent switches (or 
 > create alternate models).
 > 
 > If the decision is made to switch to a longer cycle, the release 
 > management team recommends to switch to one year directly. That would 
 > avoid changing it again anytime soon, and synchronizing on a calendar 
 > year is much simpler to follow and communicate. We also recommend 
 > announcing the change well in advance. We currently have an opportunity 
 > of making the switch when we reach the end of the release naming 
 > alphabet, which would also greatly simplify the communications around 
 > the change.
 > 
 > Finally, it is worth mentioning the impact on the stable branch work. 
 > Releasing less often would likely impact the number of stable branches 
 > that we keep on maintaining, so that we do not go too much in the past 
 > (and hit unmaintained distributions or long-gone dependencies). We 
 > currently maintain releases for 18 months before they switch to extended 
 > maintenance, which results in between 3 and 4 releases being maintained 
 > at the same time. We'd recommend switching to maintaining one-year 
 > releases for 24 months, which would result in between 2 and 3 releases 
 > being maintained at the same time. Such a change would lead to longer 
 > maintenance for our users while reducing backporting work for our 
 > developers.

Yeah, if we switch to one-year release model then definitely we need to change the
stable support policy. For example, do we need an extended maintenance phase if we
support a release for 24 months? and if we keep the EM phase too, then important thing to note
is that EM phase is the almost same amount of work upstream developers are spending now a
days in terms of testing or backports (even though we have the agreement of reducing the effort
for EM stables when needed, but I do not see that is happening, and we end up doing
the same amount of maintenance there as we do for supported stables).

As the yearly release model extend the stable support window and with our current situation of
stable team shrinking, it is an open question for us whether we as a community will be able to support
the new stable release window or not?

Another point we need to consider is how it will impact the contribution support from the companies
and volunteer contributors (we might not have many volunteer contributors now, so we can
ignore it, but let's consider companies' support). For example, the foundation membership contract
does not have contribution requirements, so companies' contribution support is always a volunteer or
based on their customer needs. In that case, we need to think about how we can keep that without any
impact. For example, change the foundation membership requirement or get companies' feedback if it
does not impact their contribution support policy.


-gmann

 > 
 > -- 
 > Thierry Carrez (ttx)
 > On behalf of the OpenStack Release Management team
 > 
 >