Re: [all][stable][ptl] Propose to EOL Rocky series
Hi,
thanks for all the feedbacks from teams so far!
@Zigo: Extended Maintenance process was created just for the same situation: to give space to interested parties to cooperate and keep things maintained even when stable releases are over their 'supported' lifetime. So it's good to see that there is interest in it! Unfortunately, with very old branches we've reached the state where gates can't be maintained and without a functional gate it's not safe to merge patches (yes, even security fixes) and they are just using resources (CI & maintainers' time). When gate is broken in such extent, then i think the community have to accept that it is not possible to merge patches confidently and needs to EOL that release. (Yes, it means that some vendors still have to do it for their own sake, testing it by their own, etc). Keeping things open in such state gives the false signal to vendors that 'it is maintained' however this is done without fully testing the changes.
Another aspect is that code cannot be cleaned up until those old branches are still present (CI jobs, project configurations, etc) which gives pain for developers.
So, however some vendors would appreciate probably to keep things open forever, for the community this is not beneficial and doable I think. Seeing that teams want to EOL their old branches is a proof that this is needed (unfortunately).
Thanks,
Előd Illés irc: elodilles @ #openstack-stable / #openstack-release
________________________________ From: Thomas Goirand thomas@goirand.fr Sent: Saturday, January 28, 2023 2:25 PM Cc: openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org Subject: Re: [all][stable][ptl] Propose to EOL Rocky series
Hi,
I understand the gate is broken, however, Rocky is in Debian LTS, and I would like to keep the possibility to merge patches, even with the gate tests disabled. CVE-2022-47951 is an example why this is important...
Thomas Goirand (zigo)
On Jan 28, 2023 11:54, Radosław Piliszek radoslaw.piliszek@gmail.com wrote:
Masakari is happy to EOL Rocky too.
Radek -yoctozepto
On Fri, 27 Jan 2023 at 20:59, Jay Faulkner jay@gr-oss.io wrote:
Thanks for doing all this cleanup work Elod. Ironic is OK with retirements of these shared resources up to Train.
Jay Faulkner
On Fri, Jan 27, 2023 at 10:12 AM Elõd Illés elod.illes@est.tech wrote:
Hi,
Similarly like the Queens branch EOL proposal [1] I would like to propose to transition every project's stable/rocky to End of Life:
- gates are mostly broken
- minimal number of activity can be seen on this branch
- some core projects already transitioned their stable/rocky to EOL recently (like ironic, neutron, nova)
- gate job definitions are still using the old, legacy zuul syntax
- gate jobs are based on Ubuntu Xenial, which is also beyond its public maintenance window date and hard to maintain
Based on the above, if there won't be any project who wants to keep open their stable/rocky, then I'll start the process of EOL'ing Rocky stable series as a whole. If anyone has any objection then please respond to this mail.
Thanks,
Előd Illés irc: elodilles @ #openstack-stable / #openstack-release
[1] https://lists.openstack.org/pipermail/openstack-discuss/2022-October/031030....
On 2/10/23 18:26, Elõd Illés wrote:
Hi,
thanks for all the feedbacks from teams so far!
@Zigo: Extended Maintenance process was created just for the same situation: to give space to interested parties to cooperate and keep things maintained even when stable releases are over their 'supported' lifetime. So it's good to see that there is interest in it! Unfortunately, with very old branches we've reached the state where gates can't be maintained and without a functional gate it's not safe to merge patches (yes, even security fixes) and they are just using resources (CI & maintainers' time). When gate is broken in such extent, then i think the community have to accept that it is not possible to merge patches confidently and needs to EOL that release.
That's where I don't agree. There are ways, outside of the OpenStack gate, to test things, in such ways that merging patches there can be a thing.
Another aspect is that code cannot be cleaned up until those old branches are still present (CI jobs, project configurations, etc) which gives pain for developers.
Just disable gating completely then.
So, however some vendors would appreciate probably to keep things open forever, for the community this is not beneficial and doable I think.
I don't agree. We need a place to share patches between distros. The official Git feels like the natural place to do so, even without any type of gating.
BTW, my Nova patches for CVE-2022-47951 in Rocky, Stein & Train are currently wrong and need another approach. I was thinking about simply disabling .vmdk altogether (rather than having a complicated code to check for the VMDK subtype). I wonder what other distros did. Where do I disucss this?
Cheers,
Thomas Goirand (zigo)
Hi,
this thread was bit of forgotten, sorry for that. A bit more than two weeks ago we had a discussion on #openstack-release about this [1].
So, to summerize, there are the issues: - stable/rocky's gate is mostly broken - more than one third of the repositories have transitioned their stable/rocky branch to EOL (including multiple core component) - old, unmaintained CI jobs, testing environments, hinders refactoring of Zuul jobs and other configurations
On the other hand, as Thomas mentioned, there is the need for some to be able to cooperate (as an example: recent security issue [2], mentioned in previous mail or in our IRC discussion) on a common place, namely in gerrit. This was originally the intention with Extended Maintenance. We just haven't thought about eternity :)
It seems that teams feel that if a branch is 'open' and in 'Extended Maintenance' then it still means it is 'fully supported', thus cannot let the gate failing AND don't want to merge patches without gate tests, that's one reason why teams rather EOL their branches.
We might need to think more about what is the best way forward.
[1] https://meetings.opendev.org/irclogs/%23openstack-release/%23openstack-relea... [2] https://security.openstack.org/ossa/OSSA-2023-002.html
Előd irc: elodilles
________________________________ From: Thomas Goirand zigo@debian.org Sent: Tuesday, February 14, 2023 7:31 PM To: Elõd Illés elod.illes@est.tech Cc: openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org Subject: Re: [all][stable][ptl] Propose to EOL Rocky series
On 2/10/23 18:26, Elõd Illés wrote:
Hi,
thanks for all the feedbacks from teams so far!
@Zigo: Extended Maintenance process was created just for the same situation: to give space to interested parties to cooperate and keep things maintained even when stable releases are over their 'supported' lifetime. So it's good to see that there is interest in it! Unfortunately, with very old branches we've reached the state where gates can't be maintained and without a functional gate it's not safe to merge patches (yes, even security fixes) and they are just using resources (CI & maintainers' time). When gate is broken in such extent, then i think the community have to accept that it is not possible to merge patches confidently and needs to EOL that release.
That's where I don't agree. There are ways, outside of the OpenStack gate, to test things, in such ways that merging patches there can be a thing.
Another aspect is that code cannot be cleaned up until those old branches are still present (CI jobs, project configurations, etc) which gives pain for developers.
Just disable gating completely then.
So, however some vendors would appreciate probably to keep things open forever, for the community this is not beneficial and doable I think.
I don't agree. We need a place to share patches between distros. The official Git feels like the natural place to do so, even without any type of gating.
BTW, my Nova patches for CVE-2022-47951 in Rocky, Stein & Train are currently wrong and need another approach. I was thinking about simply disabling .vmdk altogether (rather than having a complicated code to check for the VMDK subtype). I wonder what other distros did. Where do I disucss this?
Cheers,
Thomas Goirand (zigo)
On 3/28/23 2:34 AM, Elõd Illés wrote:
Hi,
this thread was bit of forgotten, sorry for that. A bit more than two weeks ago we had a discussion on #openstack-release about this [1].
So, to summerize, there are the issues:
- stable/rocky's gate is mostly broken
- more than one third of the repositories have transitioned their
stable/rocky branch to EOL (including multiple core component)
- old, unmaintained CI jobs, testing environments, hinders refactoring
of Zuul jobs and other configurations
On the other hand, as Thomas mentioned, there is the need for some to be able to cooperate (as an example: recent security issue [2], mentioned in previous mail or in our IRC discussion) on a common place, namely in gerrit.
The Cinder project team discussed this at the 2023.2 Bobcat vPTG and decided that we will definitely EOL rocky AND stein. Given that other core projects have already done this, they've outlived their usefulness, even if they were only to be used to discuss patches for those branches in gerrit. If a cinder patch requires patches in other services that have EOL'd that branch, that other service's patches can't be discussed in gerrit. (And if there's an alternative place to discuss, for example, a nova rocky patch, then that alternative place can be used to discuss cinder EOL patches, too.)
This was originally the intention with Extended Maintenance. We just haven't thought about eternity :)
We are also concerned about the eternal nature of older branches. There are some open source projects where the convention has been to never delete a branch, even though at a certain point, no changes will be accepted into such a branch. OpenStack, however, is not such a project, and we *do* delete branches. There's a feeling among the project team that while a branch exists, we are still responsible for it, and there's an expectation that (no matter what the EM docs say), the project team is keeping an eye on it. So we support the current declare-EOL-and-delete-the-branch policy.
Additionally, if we leave a branch in EM but turn off all CI because the team has no intention of ever merging anything into that branch, using gerrit for discussion of patches is probably unworkable as you start to get a bunch of patches into merge conflict, and it looks like the whole situation will just become a mess.
It seems that teams feel that if a branch is 'open' and in 'Extended Maintenance' then it still means it is 'fully supported', thus cannot let the gate failing AND don't want to merge patches without gate tests, that's one reason why teams rather EOL their branches.
We agree. "Not fully supported but still maintained" is a weird state. The way OpenStack has signaled "no more" for a branch is to delete it, so we continue to support the eol-and-delete policy.
We might need to think more about what is the best way forward.
We think that rocky and stein are done deals because some projects have already deleted them. The next issue is train ... is it even responsible to have a branch that supports Python 2 be in extended maintenance? We agree that this requires some more thought.
[1] https://meetings.opendev.org/irclogs/%23openstack-release/%23openstack-relea... [2] https://security.openstack.org/ossa/OSSA-2023-002.html
Előd irc: elodilles
*From:* Thomas Goirand zigo@debian.org *Sent:* Tuesday, February 14, 2023 7:31 PM *To:* Elõd Illés elod.illes@est.tech *Cc:* openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org *Subject:* Re: [all][stable][ptl] Propose to EOL Rocky series On 2/10/23 18:26, Elõd Illés wrote:
Hi,
thanks for all the feedbacks from teams so far!
@Zigo: Extended Maintenance process was created just for the same situation: to give space to interested parties to cooperate and keep things maintained even when stable releases are over their 'supported' lifetime. So it's good to see that there is interest in it! Unfortunately, with very old branches we've reached the state where gates can't be maintained and without a functional gate it's not safe to merge patches (yes, even security fixes) and they are just using resources (CI & maintainers' time). When gate is broken in such extent, then i think the community have to accept that it is not possible to merge patches confidently and needs to EOL that release.
That's where I don't agree. There are ways, outside of the OpenStack gate, to test things, in such ways that merging patches there can be a thing.
Another aspect is that code cannot be cleaned up until those old branches are still present (CI jobs, project configurations, etc) which gives pain for developers.
Just disable gating completely then.
So, however some vendors would appreciate probably to keep things open forever, for the community this is not beneficial and doable I think.
I don't agree. We need a place to share patches between distros. The official Git feels like the natural place to do so, even without any type of gating.
BTW, my Nova patches for CVE-2022-47951 in Rocky, Stein & Train are currently wrong and need another approach. I was thinking about simply disabling .vmdk altogether (rather than having a complicated code to check for the VMDK subtype). I wonder what other distros did. Where do I disucss this?
Cheers,
Thomas Goirand (zigo)
Hi,
Thanks, Brian, for sharing Cinder team's view on this topic! I was thinking what to do now, and, because I agree that Rocky (and Stein, too) is probably a done business already, I'm planning to propose the rocky-eol patches for projects that are still open, and teams can signal their decision on those patches (such as: +1 to EOL their stable/rocky branch, or -1 to keep it open).
Thanks again for the responses,
Előd ________________________________ From: Brian Rosmaita rosmaita.fossdev@gmail.com Sent: Wednesday, April 12, 2023 6:49 PM To: openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org Subject: Re: [all][stable][ptl] Propose to EOL Rocky series
On 3/28/23 2:34 AM, Elõd Illés wrote:
Hi,
this thread was bit of forgotten, sorry for that. A bit more than two weeks ago we had a discussion on #openstack-release about this [1].
So, to summerize, there are the issues:
- stable/rocky's gate is mostly broken
- more than one third of the repositories have transitioned their stable/rocky branch to EOL (including multiple core component)
- old, unmaintained CI jobs, testing environments, hinders refactoring of Zuul jobs and other configurations
On the other hand, as Thomas mentioned, there is the need for some to be able to cooperate (as an example: recent security issue [2], mentioned in previous mail or in our IRC discussion) on a common place, namely in gerrit.
The Cinder project team discussed this at the 2023.2 Bobcat vPTG and decided that we will definitely EOL rocky AND stein. Given that other core projects have already done this, they've outlived their usefulness, even if they were only to be used to discuss patches for those branches in gerrit. If a cinder patch requires patches in other services that have EOL'd that branch, that other service's patches can't be discussed in gerrit. (And if there's an alternative place to discuss, for example, a nova rocky patch, then that alternative place can be used to discuss cinder EOL patches, too.)
This was originally the intention with Extended Maintenance. We just haven't thought about eternity :)
We are also concerned about the eternal nature of older branches. There are some open source projects where the convention has been to never delete a branch, even though at a certain point, no changes will be accepted into such a branch. OpenStack, however, is not such a project, and we *do* delete branches. There's a feeling among the project team that while a branch exists, we are still responsible for it, and there's an expectation that (no matter what the EM docs say), the project team is keeping an eye on it. So we support the current declare-EOL-and-delete-the-branch policy.
Additionally, if we leave a branch in EM but turn off all CI because the team has no intention of ever merging anything into that branch, using gerrit for discussion of patches is probably unworkable as you start to get a bunch of patches into merge conflict, and it looks like the whole situation will just become a mess.
It seems that teams feel that if a branch is 'open' and in 'Extended Maintenance' then it still means it is 'fully supported', thus cannot let the gate failing AND don't want to merge patches without gate tests, that's one reason why teams rather EOL their branches.
We agree. "Not fully supported but still maintained" is a weird state. The way OpenStack has signaled "no more" for a branch is to delete it, so we continue to support the eol-and-delete policy.
We might need to think more about what is the best way forward.
We think that rocky and stein are done deals because some projects have already deleted them. The next issue is train ... is it even responsible to have a branch that supports Python 2 be in extended maintenance? We agree that this requires some more thought.
[1] https://meetings.opendev.org/irclogs/%23openstack-release/%23openstack-relea... [2] https://security.openstack.org/ossa/OSSA-2023-002.html
Előd irc: elodilles
*From:* Thomas Goirand zigo@debian.org *Sent:* Tuesday, February 14, 2023 7:31 PM *To:* Elõd Illés elod.illes@est.tech *Cc:* openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org *Subject:* Re: [all][stable][ptl] Propose to EOL Rocky series On 2/10/23 18:26, Elõd Illés wrote:
Hi,
thanks for all the feedbacks from teams so far!
@Zigo: Extended Maintenance process was created just for the same situation: to give space to interested parties to cooperate and keep things maintained even when stable releases are over their 'supported' lifetime. So it's good to see that there is interest in it! Unfortunately, with very old branches we've reached the state where gates can't be maintained and without a functional gate it's not safe to merge patches (yes, even security fixes) and they are just using resources (CI & maintainers' time). When gate is broken in such extent, then i think the community have to accept that it is not possible to merge patches confidently and needs to EOL that release.
That's where I don't agree. There are ways, outside of the OpenStack gate, to test things, in such ways that merging patches there can be a thing.
Another aspect is that code cannot be cleaned up until those old branches are still present (CI jobs, project configurations, etc) which gives pain for developers.
Just disable gating completely then.
So, however some vendors would appreciate probably to keep things open forever, for the community this is not beneficial and doable I think.
I don't agree. We need a place to share patches between distros. The official Git feels like the natural place to do so, even without any type of gating.
BTW, my Nova patches for CVE-2022-47951 in Rocky, Stein & Train are currently wrong and need another approach. I was thinking about simply disabling .vmdk altogether (rather than having a complicated code to check for the VMDK subtype). I wonder what other distros did. Where do I disucss this?
Cheers,
Thomas Goirand (zigo)
participants (3)
-
Brian Rosmaita
-
Elõd Illés
-
Thomas Goirand