[TC][PTL][ops][all] Release cadence, deprecation, and testing
Hi all,
As you probably already know from gmann's reporting, the TC recently passed a resolution[1] regarding the release cadence. This, in response to some discussion around moving to different release cycle lengths, becomes mostly a change regarding our deprecation procedures, as well as a call to support the ability to skip certain releases (designated "tock" releases) while sticking on a yearly cycle (designated "tick" releases).
In support of the goal of moving to this new cadence for the AA release (i.e. the yet unnamed release to come after Zed), I have introduced[2] a new grenade-skip-level job, which tests skipping "tock" releases and going straight from one tick to the following. We intend this to be a voting job in AA, making sure it works from Yoga->AA. Right now it tests Wallaby->Yoga for informational purposes. We would recommend projects enable that as voting, or define their own job to do similar testing as soon as possible. Manila has already started[3] this. Note that existing grenade testing between each adjacent release remains unchanged in this scheme.
On the topic of deprecation, we moved the current definition of deprecation procedures from the soon-to-be-removed tag definition to a new document in project-team-guide[4]. This now describes the procedures in terms of the tick-tock release arrangement, and is worth a read.
As the PTG is coming up, we will certainly discuss this topic in the PTL Interaction[5] session. It would also be advisable to discuss the project-specific details of this in the various project sessions. By identifying this plan early and setting our goals for AA, we have a good amount of time between now and then to get our heads wrapped around what (if any) things we need to be doing. Luckily, the introduction of the job to test this from wallaby to yoga required almost no project changes to get working, which is a good indicator that it won't be too laborious for most projects to adopt.
--Dan
1: https://governance.openstack.org/tc/resolutions/20220210-release-cadence-adj... 2: https://review.opendev.org/c/openstack/grenade/+/826101 3: https://review.opendev.org/c/openstack/manila/+/830277 4: https://docs.openstack.org/project-team-guide/deprecation.html 5: https://etherpad.opendev.org/p/tc-ptl-interaction-zed
On Wed, 9 Mar 2022 at 22:35, Dan Smith dms@danplanet.com wrote:
Hi all,
As you probably already know from gmann's reporting, the TC recently passed a resolution[1] regarding the release cadence. This, in response to some discussion around moving to different release cycle lengths, becomes mostly a change regarding our deprecation procedures, as well as a call to support the ability to skip certain releases (designated "tock" releases) while sticking on a yearly cycle (designated "tick" releases).
In support of the goal of moving to this new cadence for the AA release (i.e. the yet unnamed release to come after Zed), I have introduced[2] a new grenade-skip-level job, which tests skipping "tock" releases and going straight from one tick to the following. We intend this to be a voting job in AA, making sure it works from Yoga->AA. Right now it tests Wallaby->Yoga for informational purposes. We would recommend projects enable that as voting, or define their own job to do similar testing as soon as possible. Manila has already started[3] this. Note that existing grenade testing between each adjacent release remains unchanged in this scheme.
On the topic of deprecation, we moved the current definition of deprecation procedures from the soon-to-be-removed tag definition to a new document in project-team-guide[4]. This now describes the procedures in terms of the tick-tock release arrangement, and is worth a read.
As the PTG is coming up, we will certainly discuss this topic in the PTL Interaction[5] session. It would also be advisable to discuss the project-specific details of this in the various project sessions. By identifying this plan early and setting our goals for AA, we have a good amount of time between now and then to get our heads wrapped around what (if any) things we need to be doing. Luckily, the introduction of the job to test this from wallaby to yoga required almost no project changes to get working, which is a good indicator that it won't be too laborious for most projects to adopt.
--Dan
1: https://governance.openstack.org/tc/resolutions/20220210-release-cadence-adj... 2: https://review.opendev.org/c/openstack/grenade/+/826101 3: https://review.opendev.org/c/openstack/manila/+/830277 4: https://docs.openstack.org/project-team-guide/deprecation.html 5: https://etherpad.opendev.org/p/tc-ptl-interaction-zed
Thanks for writing this up, Dan. Overall it seems sensible. One thing that jumps out from the TC resolution is this:
"This scheme does not necessarily dictate that live or rolling upgrades need to be supported between tick releases. Meaning RPC compatibility between N to N-1 guarantees can remain, resulting in deployments that are on a tick-tick release schedule requiring some downtime during an upgrade because components will be spanning more than two actual releases."
From the perspective of deployment tools, this would mean needing to
support two different upgrade strategies. Given the constraints around DB migrations, would it be too much effort for projects to also support N-2 to N RPC compatibility?
Mark
"This scheme does not necessarily dictate that live or rolling upgrades need to be supported between tick releases. Meaning RPC compatibility between N to N-1 guarantees can remain, resulting in deployments that are on a tick-tick release schedule requiring some downtime during an upgrade because components will be spanning more than two actual releases."
From the perspective of deployment tools, this would mean needing to support two different upgrade strategies. Given the constraints around DB migrations, would it be too much effort for projects to also support N-2 to N RPC compatibility?
I can tell you it's much harder to do it for RPC than DB migrations. I personally think it's too much to ask projects to move to that right away, even though some may be able to do it now. It also has farther reaches than you might expect, such as dependent components that change over time (think OVS/OVN, libvirt, qemu).
It may just be the early morning and lack of caffeination, but I'm not sure what the deployment concern is exactly. You rely entirely on live rolling upgrades today? Presumably only for some of the services?
--Dan
On Thu, 10 Mar 2022 at 14:50, Dan Smith dms@danplanet.com wrote:
"This scheme does not necessarily dictate that live or rolling upgrades need to be supported between tick releases. Meaning RPC compatibility between N to N-1 guarantees can remain, resulting in deployments that are on a tick-tick release schedule requiring some downtime during an upgrade because components will be spanning more than two actual releases."
From the perspective of deployment tools, this would mean needing to support two different upgrade strategies. Given the constraints around DB migrations, would it be too much effort for projects to also support N-2 to N RPC compatibility?
I can tell you it's much harder to do it for RPC than DB migrations. I personally think it's too much to ask projects to move to that right away, even though some may be able to do it now. It also has farther reaches than you might expect, such as dependent components that change over time (think OVS/OVN, libvirt, qemu).
It may just be the early morning and lack of caffeination, but I'm not sure what the deployment concern is exactly. You rely entirely on live rolling upgrades today? Presumably only for some of the services?
Kolla Ansible uses rolling upgrades for most services. It doesn't support full stop upgrades. I expect it wouldn't be too difficult to implement, but it would be a shame to have two different upgrade paths.
--Dan
Kolla Ansible uses rolling upgrades for most services. It doesn't support full stop upgrades. I expect it wouldn't be too difficult to implement, but it would be a shame to have two different upgrade paths.
Okay, well, our method for testing RPC compatibility is grenade multinode, so we'd need a multinode job that does the skip level piece. I could try poking at that to see, but again, I think it's asking a lot from the projects to make that leap right away. If I'm wrong about that, then we could surely explore that route.
--Dan
On Wed, Mar 9, 2022 at 11:36 PM Dan Smith dms@danplanet.com wrote:
Hi all,
As you probably already know from gmann's reporting, the TC recently passed a resolution[1] regarding the release cadence. This, in response to some discussion around moving to different release cycle lengths, becomes mostly a change regarding our deprecation procedures, as well as a call to support the ability to skip certain releases (designated "tock" releases) while sticking on a yearly cycle (designated "tick" releases).
In support of the goal of moving to this new cadence for the AA release (i.e. the yet unnamed release to come after Zed), I have introduced[2] a new grenade-skip-level job, which tests skipping "tock" releases and going straight from one tick to the following. We intend this to be a voting job in AA, making sure it works from Yoga->AA. Right now it tests Wallaby->Yoga for informational purposes. We would recommend projects enable that as voting, or define their own job to do similar testing as soon as possible. Manila has already started[3] this. Note that existing grenade testing between each adjacent release remains unchanged in this scheme.
I don't think we have cycles in Ironic to do that. Unless somebody who really cares wants to not just do the work on setting up such jobs, but also maintaining them long-term.
On the topic of deprecation, we moved the current definition of deprecation procedures from the soon-to-be-removed tag definition to a new document in project-team-guide[4]. This now describes the procedures in terms of the tick-tock release arrangement, and is worth a read.
Same. I don't want to keep deprecated features for a year.
Dmitry
As the PTG is coming up, we will certainly discuss this topic in the PTL Interaction[5] session. It would also be advisable to discuss the project-specific details of this in the various project sessions. By identifying this plan early and setting our goals for AA, we have a good amount of time between now and then to get our heads wrapped around what (if any) things we need to be doing. Luckily, the introduction of the job to test this from wallaby to yoga required almost no project changes to get working, which is a good indicator that it won't be too laborious for most projects to adopt.
--Dan
1: https://governance.openstack.org/tc/resolutions/20220210-release-cadence-adj... 2: https://review.opendev.org/c/openstack/grenade/+/826101 3: https://review.opendev.org/c/openstack/manila/+/830277 4: https://docs.openstack.org/project-team-guide/deprecation.html 5: https://etherpad.opendev.org/p/tc-ptl-interaction-zed
participants (3)
-
Dan Smith
-
Dmitry Tantsur
-
Mark Goddard