Release Cycle Observations
As many of you know I am fairly new to public participation in Openstack, but I am a long time lurker and operator. This is my first cycle and I wanted to capture an observation on the release cycle of Openstack. *this is only an observation for discussion*
Over the last week or so I have watch people just about kill themselves to get features into Openstack for this release. Everyone in this community appears to be giving it their all. And way to go everyone for pushing so hard. Seriously!
So this brings me to my observation.
Can the release cycle be modified to ease this push at the end? I think what we all want is a smooth and steady stream of features and bug fixes, but a 6 month release cycle seems to do directly the opposite. Also if everyone has gate issues at the end of the cycle, is it the gate or the cycle? The gate seemed to work fine early on...
Maybe if we went to a Major / Minor release sort of system we could remove some burden to *get that feature into X release* and we could steady out that flow of features / bug fixes. Maybe something like a 9 month major, one month minor release cycle. It would also probably help with micro api versioning.
I am sure the more experienced developers here can highlight pros / cons, etc of all of the release types.
Just a thought. Tell me yours.
~/DonnyD
On Thu, Sep 26, 2019, at 9:38 AM, Donny Davis wrote:
As many of you know I am fairly new to public participation in Openstack, but I am a long time lurker and operator. This is my first cycle and I wanted to capture an observation on the release cycle of Openstack. *this is only an observation for discussion*
Over the last week or so I have watch people just about kill themselves to get features into Openstack for this release. Everyone in this community appears to be giving it their all. And way to go everyone for pushing so hard. Seriously!
So this brings me to my observation.
Can the release cycle be modified to ease this push at the end? I think what we all want is a smooth and steady stream of features and bug fixes, but a 6 month release cycle seems to do directly the opposite. Also if everyone has gate issues at the end of the cycle, is it the gate or the cycle? The gate seemed to work fine early on...
I think there are a few things that happen with the gate to make it seem worse at the end of the cycle. The first is that at every other point in time you can just recheck until code merges, taking your time as there is no deadline. The problems are often there but can be ignored.
Next the cost of failure goes up with larger queues. We tend to have very large queues at the end of the cycle. This means we feel the pain of resource waste more at the end of the cycle when ideally we'd avoid it entirely.
Finally despite the problems we do still merge code and often more than at any other time of the cycle. There is a lot of new code coming in last minute that shakes loose race conditions and other failures.
Maybe if we went to a Major / Minor release sort of system we could remove some burden to *get that feature into X release* and we could steady out that flow of features / bug fixes. Maybe something like a 9 month major, one month minor release cycle. It would also probably help with micro api versioning.
I am sure the more experienced developers here can highlight pros / cons, etc of all of the release types.
Just a thought. Tell me yours.
~/DonnyD
On 9/26/2019 8:38 AM, Donny Davis wrote:
Can the release cycle be modified to ease this push at the end? I think what we all want is a smooth and steady stream of features and bug fixes, but a 6 month release cycle seems to do directly the opposite. Also if everyone has gate issues at the end of the cycle, is it the gate or the cycle? The gate seemed to work fine early on...
Oh lordy, you've started *this* thread again. :P
Yes this has come up many a time and Clark replied about gate issues - they are always there but people care less because there is time to recheck earlier in the cycle.
In nova we've historically tried all sorts of different tricks to try and manipulate the schedule within the 6 month cycle, like priority vs non-priority spec and feature freezes, runways, etc, because no matter what we do, the most code and review velocity kicks in about 2 weeks before a deadline. Changing the release cycle isn't going to change the human nature about that. People are still going to wait until too late in the cycle to focus on their complicated feature, and reviewers are going to wait too long to get serious about reviewing the more complicated features. The runways concept is meant to help with that so once a blueprint is ready for review it's in a 2 week queue and contributors can expect to get core reviews, but not all features are the same or get the same interest from the core team, so it has it's pitfalls. Maybe people don't want to admit it, but this is a corporate driven project so a lot of the time money talks so people are going to spend time on what their employer wants them to spend time on, which might not be aligned with "the good of the project" or some lower priority feature that someone else wants to get in.
In last week's nova meeting we also talked about some of this [1]. If you change the release cycle to do major and minor versions, that's going to be pretty complicated for testing upgrades, because grenade goes from major to major based on branches. We could probably make it work (and maybe it's already possible, I don't know to go from tag (minor release) to master, but the point is you're going to blow up the upgrade test matrix and it's not clear anyone is even going to be consuming those minor intermediate releases. I think this is partly why the release team stopped doing a cycle-with-milestone [2] release because no one was consuming the milestone releases.
Anyway, if there is some magic bullet for reducing noise and increasing focus on what we think is most important to merge in a release I don't think we've found it yet even though we've talked about this many times for many years.
[1] http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-09-19-14.00.log.... [2] https://releases.openstack.org/reference/release_models.html#cycle-with-mile...
On 9/26/19 9:42 AM, Matt Riedemann wrote:
In last week's nova meeting we also talked about some of this [1]. If you change the release cycle to do major and minor versions, that's going to be pretty complicated for testing upgrades, because grenade goes from major to major based on branches. We could probably make it work (and maybe it's already possible, I don't know to go from tag (minor release) to master, but the point is you're going to blow up the upgrade test matrix and it's not clear anyone is even going to be consuming those minor intermediate releases. I think this is partly why the release team stopped doing a cycle-with-milestone [2] release because no one was consuming the milestone releases.
We've even found that since we started supporting FFU between certain releases that many customers won't touch the intervening releases, even with six months between them. They're basically on a year-and-a-half cycle. This actually makes things worse for the FFU releases because having a feature miss one means waiting that much longer for it to show up (or we have to carry downstream backports, which sucks).
I know we'd like to have everyone CD'ing master, but for the foreseeable future I think it's more likely that we're going to have a non-trivial number of deployers who stick to the longest release cycle that can be supported, and that's going to create added pressure during feature freeze for whatever that release is.
So would a longer release cycle help or hurt that? Would having two trains make it better or worse?
Just want to make things clear that I am only curious and wanted to share what I see.
~/D
On Thu, Sep 26, 2019 at 12:17 PM Ben Nemec openstack@nemebean.com wrote:
On 9/26/19 9:42 AM, Matt Riedemann wrote:
In last week's nova meeting we also talked about some of this [1]. If you change the release cycle to do major and minor versions, that's going to be pretty complicated for testing upgrades, because grenade goes from major to major based on branches. We could probably make it work (and maybe it's already possible, I don't know to go from tag (minor release) to master, but the point is you're going to blow up the upgrade test matrix and it's not clear anyone is even going to be consuming those minor intermediate releases. I think this is partly why the release team stopped doing a cycle-with-milestone [2] release because no one was consuming the milestone releases.
We've even found that since we started supporting FFU between certain releases that many customers won't touch the intervening releases, even with six months between them. They're basically on a year-and-a-half cycle. This actually makes things worse for the FFU releases because having a feature miss one means waiting that much longer for it to show up (or we have to carry downstream backports, which sucks).
I know we'd like to have everyone CD'ing master, but for the foreseeable future I think it's more likely that we're going to have a non-trivial number of deployers who stick to the longest release cycle that can be supported, and that's going to create added pressure during feature freeze for whatever that release is.
On 9/26/19 11:29 AM, Donny Davis wrote:
So would a longer release cycle help or hurt that? Would having two trains make it better or worse?
My prediction is that the shorter release cycle train would be largely ignored, and the feature freeze madness for the longer cycle would be even worse. That's nothing more than an educated guess on my part though.
It also may still be worth it to move to the longer cycle. On some level, the feature freeze crunch is going to happen anyway. At least that way it happens less often.
Just want to make things clear that I am only curious and wanted to share what I see.
~/D
On Thu, Sep 26, 2019 at 12:17 PM Ben Nemec <openstack@nemebean.com mailto:openstack@nemebean.com> wrote:
On 9/26/19 9:42 AM, Matt Riedemann wrote: > In last week's nova meeting we also talked about some of this [1]. If > you change the release cycle to do major and minor versions, that's > going to be pretty complicated for testing upgrades, because grenade > goes from major to major based on branches. We could probably make it > work (and maybe it's already possible, I don't know to go from tag > (minor release) to master, but the point is you're going to blow up the > upgrade test matrix and it's not clear anyone is even going to be > consuming those minor intermediate releases. I think this is partly why > the release team stopped doing a cycle-with-milestone [2] release > because no one was consuming the milestone releases. We've even found that since we started supporting FFU between certain releases that many customers won't touch the intervening releases, even with six months between them. They're basically on a year-and-a-half cycle. This actually makes things worse for the FFU releases because having a feature miss one means waiting that much longer for it to show up (or we have to carry downstream backports, which sucks). I know we'd like to have everyone CD'ing master, but for the foreseeable future I think it's more likely that we're going to have a non-trivial number of deployers who stick to the longest release cycle that can be supported, and that's going to create added pressure during feature freeze for whatever that release is.
On 9/26/2019 3:34 PM, Ben Nemec wrote:
On 9/26/19 11:29 AM, Donny Davis wrote:
So would a longer release cycle help or hurt that? Would having two trains make it better or worse?
My prediction is that the shorter release cycle train would be largely ignored, and the feature freeze madness for the longer cycle would be even worse. That's nothing more than an educated guess on my part though.
I agree with Ben. I think it is admirable to want to find a better way to do things but we have tried numerous different approaches in Cinder to try and avoid the crunch at the end but the fact is that it appears that everyone procrastinates until the last minute. It seems to be one of the problems of being an open community where people don't always have full time to work on development.
It also may still be worth it to move to the longer cycle. On some level, the feature freeze crunch is going to happen anyway. At least that way it happens less often.
Just want to make things clear that I am only curious and wanted to share what I see.
~/D
On Thu, Sep 26, 2019 at 12:17 PM Ben Nemec <openstack@nemebean.com mailto:openstack@nemebean.com> wrote:
On 9/26/19 9:42 AM, Matt Riedemann wrote: > In last week's nova meeting we also talked about some of this [1]. If > you change the release cycle to do major and minor versions, that's > going to be pretty complicated for testing upgrades, because grenade > goes from major to major based on branches. We could probably make it > work (and maybe it's already possible, I don't know to go from tag > (minor release) to master, but the point is you're going to blow up the > upgrade test matrix and it's not clear anyone is even going to be > consuming those minor intermediate releases. I think this is partly why > the release team stopped doing a cycle-with-milestone [2] release > because no one was consuming the milestone releases.
We've even found that since we started supporting FFU between certain releases that many customers won't touch the intervening releases, even with six months between them. They're basically on a year-and-a-half cycle. This actually makes things worse for the FFU releases because having a feature miss one means waiting that much longer for it to show up (or we have to carry downstream backports, which sucks).
I know we'd like to have everyone CD'ing master, but for the foreseeable future I think it's more likely that we're going to have a non-trivial number of deployers who stick to the longest release cycle that can be supported, and that's going to create added pressure during feature freeze for whatever that release is.
So when you say largely ignored, do you mean by downstream vendors who package Openstack or by public clouds who use it daily? I can see a longer release cycle being better for vendors so they don't have to do FFU to begin with and public clouds wanting faster releases so they can get features to customers sooner.
On Thu, Sep 26, 2019 at 4:34 PM Ben Nemec openstack@nemebean.com wrote:
On 9/26/19 11:29 AM, Donny Davis wrote:
So would a longer release cycle help or hurt that? Would having two trains make it better or worse?
My prediction is that the shorter release cycle train would be largely ignored, and the feature freeze madness for the longer cycle would be even worse. That's nothing more than an educated guess on my part though.
It also may still be worth it to move to the longer cycle. On some level, the feature freeze crunch is going to happen anyway. At least that way it happens less often.
Just want to make things clear that I am only curious and wanted to share what I see.
~/D
On Thu, Sep 26, 2019 at 12:17 PM Ben Nemec <openstack@nemebean.com mailto:openstack@nemebean.com> wrote:
On 9/26/19 9:42 AM, Matt Riedemann wrote: > In last week's nova meeting we also talked about some of this [1]. If > you change the release cycle to do major and minor versions,that's
> going to be pretty complicated for testing upgrades, becausegrenade
> goes from major to major based on branches. We could probably make it > work (and maybe it's already possible, I don't know to go from tag > (minor release) to master, but the point is you're going to blow up the > upgrade test matrix and it's not clear anyone is even going to be > consuming those minor intermediate releases. I think this is partly why > the release team stopped doing a cycle-with-milestone [2] release > because no one was consuming the milestone releases. We've even found that since we started supporting FFU between certain releases that many customers won't touch the intervening releases,even
with six months between them. They're basically on a year-and-a-half cycle. This actually makes things worse for the FFU releases because having a feature miss one means waiting that much longer for it toshow
up (or we have to carry downstream backports, which sucks). I know we'd like to have everyone CD'ing master, but for the foreseeable future I think it's more likely that we're going to have anon-trivial
number of deployers who stick to the longest release cycle that canbe
supported, and that's going to create added pressure during feature freeze for whatever that release is.
Donny Davis wrote:
So when you say largely ignored, do you mean by downstream vendors who package Openstack or by public clouds who use it daily? I can see a longer release cycle being better for vendors so they don't have to do FFU to begin with and public clouds wanting faster releases so they can get features to customers sooner.
The release cycle length discussion has happened roughly every 8 months for the last 9 years, and I don't think the reality has changed.
Longer release cycles does not facilitate merging features, nor does it spread the load. Evidence would rather point to the opposite (twice-bigger crunch at the end of the larger cycle).
Release cycle length actually affects three things.
1- with longer cycles, boilerplate tasks around releasing happen less often. This is largely a non-issue now that most of the release process is automated.
2- with longer cycles, the feedback loop between when a feature is written and when it's actually in use by users is getting longer. This is bad as the person who writes the feature in the first place may have moved to other things by the time the initial user feedback comes.
3- with longer cycles, there is less pressure on downstream to integrate the new release in their products. This remains true, even if we now have multiple options to skip irrelevant releases in products.
The current 6-month cycle is a trade-off, essentially a balance between (2) and (3). It is as long as we think we can stretch the cycle duration without ruining the feedback loop.
Obviously nobody is happy with it, with some wanting it to be longer, and some wanting it to be shorter. Which means that it is probably a good middle ground :)
On 9/26/19 7:17 PM, Donny Davis wrote:
So when you say largely ignored, do you mean by downstream vendors who package Openstack or by public clouds who use it daily? I can see a longer release cycle being better for vendors so they don't have to do FFU to begin with and public clouds wanting faster releases so they can get features to customers sooner.
I see FFU as our longer release cycle though. Since it's a supported path, I'm not sure it makes sense to try to work around that.
As far as public cloud, I don't have as much insight as I haven't worked directly with as many of them. I can say that as a user who at one point needed some recent features, plenty of public clouds were not running the latest release either. There were a few that did track releases pretty closely, but many were 2 or 3 behind. Admittedly, that was a couple of years ago so I don't know the current state of public clouds. I guess the OpenStack user survey probably holds the answer to that.
On Thu, Sep 26, 2019 at 4:34 PM Ben Nemec <openstack@nemebean.com mailto:openstack@nemebean.com> wrote:
On 9/26/19 11:29 AM, Donny Davis wrote: > So would a longer release cycle help or hurt that? Would having two > trains make it better or worse? My prediction is that the shorter release cycle train would be largely ignored, and the feature freeze madness for the longer cycle would be even worse. That's nothing more than an educated guess on my part though. It also may still be worth it to move to the longer cycle. On some level, the feature freeze crunch is going to happen anyway. At least that way it happens less often. > > Just want to make things clear that I am only curious and wanted to > share what I see. > > ~/D > > On Thu, Sep 26, 2019 at 12:17 PM Ben Nemec <openstack@nemebean.com <mailto:openstack@nemebean.com> > <mailto:openstack@nemebean.com <mailto:openstack@nemebean.com>>> wrote: > > > > On 9/26/19 9:42 AM, Matt Riedemann wrote: > > In last week's nova meeting we also talked about some of this > [1]. If > > you change the release cycle to do major and minor versions, that's > > going to be pretty complicated for testing upgrades, because grenade > > goes from major to major based on branches. We could probably > make it > > work (and maybe it's already possible, I don't know to go from tag > > (minor release) to master, but the point is you're going to blow > up the > > upgrade test matrix and it's not clear anyone is even going to be > > consuming those minor intermediate releases. I think this is > partly why > > the release team stopped doing a cycle-with-milestone [2] release > > because no one was consuming the milestone releases. > > We've even found that since we started supporting FFU between certain > releases that many customers won't touch the intervening releases, even > with six months between them. They're basically on a year-and-a-half > cycle. This actually makes things worse for the FFU releases because > having a feature miss one means waiting that much longer for it to show > up (or we have to carry downstream backports, which sucks). > > I know we'd like to have everyone CD'ing master, but for the > foreseeable > future I think it's more likely that we're going to have a non-trivial > number of deployers who stick to the longest release cycle that can be > supported, and that's going to create added pressure during feature > freeze for whatever that release is. >
On 9/27/2019 11:08 AM, Ben Nemec wrote:
As far as public cloud, I don't have as much insight as I haven't worked directly with as many of them. I can say that as a user who at one point needed some recent features, plenty of public clouds were not running the latest release either. There were a few that did track releases pretty closely, but many were 2 or 3 behind. Admittedly, that was a couple of years ago so I don't know the current state of public clouds. I guess the OpenStack user survey probably holds the answer to that.
VEXXHOST is consistently on the current release as soon as it's out which is awesome.
On 9/27/19 11:13 AM, Matt Riedemann wrote:
On 9/27/2019 11:08 AM, Ben Nemec wrote:
As far as public cloud, I don't have as much insight as I haven't worked directly with as many of them. I can say that as a user who at one point needed some recent features, plenty of public clouds were not running the latest release either. There were a few that did track releases pretty closely, but many were 2 or 3 behind. Admittedly, that was a couple of years ago so I don't know the current state of public clouds. I guess the OpenStack user survey probably holds the answer to that.
VEXXHOST is consistently on the current release as soon as it's out which is awesome.
Yep, and that's where I ultimately landed for the work I was referring to. :-)
On 9/26/19 2:51 PM, Sean McGinnis wrote:
I know we'd like to have everyone CD'ing master
Watch who you're lumping in with the "we" statement. ;)
Heh, fair. I thought about qualifying that, but decided by the time I got done listing the caveats I'd have forgotten my original point. :-)
On 9/26/19 9:51 PM, Sean McGinnis wrote:
I know we'd like to have everyone CD'ing master
Watch who you're lumping in with the "we" statement. ;)
You've pinpointed what the problem is.
Everyone but OpenStack upstream would like to stop having to upgrade every 6 months. The only way this could be resolved would be an OpenStack LTS release let's say every 2 years, and allowing upgrade between them, though that's probably too much effort upstream. We have different groups wishing for the opposite thing to happen.
I don't see this problem going away anytime soon.
Cheers,
Thomas Goirand (zigo)
On Fri, Sep 27, 2019 at 10:47 PM Thomas Goirand zigo@debian.org wrote:
On 9/26/19 9:51 PM, Sean McGinnis wrote:
I know we'd like to have everyone CD'ing master
Watch who you're lumping in with the "we" statement. ;)
You've pinpointed what the problem is.
Everyone but OpenStack upstream would like to stop having to upgrade every 6 months.
Yep, but the same "everyone" want to have features now or better yesterday, not in 2-3 years ;)
The only way this could be resolved would be an OpenStack LTS release let's say every 2 years, and allowing upgrade between them, though that's probably too much effort upstream. We have different groups wishing for the opposite thing to happen.
I don't see this problem going away anytime soon.
Cheers,
Thomas Goirand (zigo)
On 10/1/19 12:05 PM, Dmitry Tantsur wrote:
On Fri, Sep 27, 2019 at 10:47 PM Thomas Goirand <zigo@debian.org mailto:zigo@debian.org> wrote:
On 9/26/19 9:51 PM, Sean McGinnis wrote: >> I know we'd like to have everyone CD'ing master > > Watch who you're lumping in with the "we" statement. ;) You've pinpointed what the problem is. Everyone but OpenStack upstream would like to stop having to upgrade every 6 months.Yep, but the same "everyone" want to have features now or better yesterday, not in 2-3 years ;)
This probably was the case a few years ago, when OpenStack was young. Now that it has matured, and has all the needed features, things have changed a lot.
Thomas
On Wed, Oct 2, 2019 at 10:31 AM Thomas Goirand zigo@debian.org wrote:
On 10/1/19 12:05 PM, Dmitry Tantsur wrote:
On Fri, Sep 27, 2019 at 10:47 PM Thomas Goirand <zigo@debian.org mailto:zigo@debian.org> wrote:
On 9/26/19 9:51 PM, Sean McGinnis wrote: >> I know we'd like to have everyone CD'ing master > > Watch who you're lumping in with the "we" statement. ;) You've pinpointed what the problem is. Everyone but OpenStack upstream would like to stop having to upgrade every 6 months.Yep, but the same "everyone" want to have features now or better yesterday, not in 2-3 years ;)
This probably was the case a few years ago, when OpenStack was young. Now that it has matured, and has all the needed features, things have changed a lot.
This is still the case often enough in my world. IPv6 comes to mind as an example.
Thomas
On Wed, 2019-10-09 at 12:04 +0200, Dmitry Tantsur wrote:
On Wed, Oct 2, 2019 at 10:31 AM Thomas Goirand zigo@debian.org wrote:
On 10/1/19 12:05 PM, Dmitry Tantsur wrote:
On Fri, Sep 27, 2019 at 10:47 PM Thomas Goirand <zigo@debian.org mailto:zigo@debian.org> wrote:
On 9/26/19 9:51 PM, Sean McGinnis wrote: >> I know we'd like to have everyone CD'ing master > > Watch who you're lumping in with the "we" statement. ;) You've pinpointed what the problem is. Everyone but OpenStack upstream would like to stop having to upgrade every 6 months.
im not sure that is true. i think if upgrades where as easy as a yum update or apt upgrade people would not mind 6 month or shorter upgrade cycle but even though tooling has imporoved we are a long way from upgrades being trivial.
Yep, but the same "everyone" want to have features now or better yesterday, not in 2-3 years ;)
yes and this is a double edge sword in more ways then one. we have a large proportion of our customer base that are only now upgrading to queens from Newton. so they are already running a 2-3 year out of date openstack and when they upgrade they would also like all the features that were only added in train backported to Queens which is our current LTS donwstream.
Our internal data on deployments more or less shows that most non lts releases downstream are ignored by larger customers createing a pressure to backport features that we cant resonably do given our current tooling and desire to not create a large fork.
This probably was the case a few years ago, when OpenStack was young. Now that it has matured, and has all the needed features, things have changed a lot.
i dont think it has. i think many of the need feature are now avaiable in master although looking at our downstream back log there are also a lot of feature that are not avilable. the issue is that because upgrading has been so painful for many for so long they are not willing in many case to go to the latest release. maybe in another 2 years time this statement will be more correct as the majority of clouds will be running stien+(i hope).
This is still the case often enough in my world. IPv6 comes to mind as an example.
Thomas
participants (10)
-
Ben Nemec
-
Clark Boylan
-
Dmitry Tantsur
-
Donny Davis
-
Jay Bryant
-
Matt Riedemann
-
Sean McGinnis
-
Sean Mooney
-
Thierry Carrez
-
Thomas Goirand