[all] Keeping old unmaintained branches open for longer time, but with no CI, instead of EOL tags
Hi, It's our policy that we maintain only a few branches, currently, from Caracal to Epoxy (so, 3 stable branches). Currently, when a Stable branch becomes unsuported, we rename that branch as: unmaintained/2023.1 which is fine, as we can still commit there. Though after some times, the branches are deleted, and instead, we just do a tag such as: zed-EOL Jeremy Stanley just wrote to me on IRC: "workflows, processes and policies are built around many years of an assumption that unused branches will be deleted" Keeping unmaintained branches "has put a lot of additional strain on project maintainers and our systems" (still quoting Jeremy). Though each time there's a new (grave?) security problem, this topic comes back, as we still have no solution for these cases. As a downstream, I still need to write backports for these important patches, sometimes long after the branches are EOL. Red Hat too... While I agree with all what Jeremy told me on IRC, and I don't think we should attempt to maintain the CI for too long (as this takes too much of our time), I still believe we should keep a way to share patches. Downstream (like myself for Debian) run their own CI based on packages, and it's always better if we can share patches. It doesn't have to be with a maintained CI. Just having opened branches would be enough. Your thoughts? Cheers, Thomas Goirand (zigo)
I'm going to completely remove any logistical concerns from this reply; I'm going to assume it's possible because I don't think we should do it in any case. On 8/21/25 8:05 AM, Thomas Goirand wrote:
Hi,
It's our policy that we maintain only a few branches, currently, from Caracal to Epoxy (so, 3 stable branches).
Currently, when a Stable branch becomes unsuported, we rename that branch as:
unmaintained/2023.1
which is fine, as we can still commit there. Though after some times, the branches are deleted, and instead, we just do a tag such as:
zed-EOL
This is a good, positive signal to our customers, and their bosses, that it's not appropriate to run these versions anymore. I personally know of multiple clusters where 'but the branch is still open' was used as a justification to not-upgrade. I have no interest in enabling these use cases because I think it's actively harmful to the OpenStack ecosystem for people to be on dramatically old releases. I'll note this opinion is strongly influenced by the fact I work on a portion of OpenStack (Ironic) for which new releases are not just adding features, but adding and improving hardware support. Keeping these branches open, even with unmaintained/ terminology, at a certain point tells people that other folks are using years-old OpenStack, and that maybe it's OK for them to as well. I don't want to send this message.
Jeremy Stanley just wrote to me on IRC:
"workflows, processes and policies are built around many years of an assumption that unused branches will be deleted"
Keeping unmaintained branches "has put a lot of additional strain on project maintainers and our systems" (still quoting Jeremy).
Though each time there's a new (grave?) security problem, this topic comes back, as we still have no solution for these cases. As a downstream, I still need to write backports for these important patches, sometimes long after the branches are EOL. Red Hat too...
There have been multiple OSSAs over the last two years. I've done an (admittedly quick) review of the bugs associated with those OSSAs -- you were the only one requesting or mentioning support for EOL releases in those bugs. I know you may represent multiple downstream Debian customers; but I don't agree that this is a recurring major issue in the community. I'll also note that on those bugs, if someone wanted to provide older patches they could attach them there. Given the low-occurrence of CVE-worthy vulnerabilities (6 since the start of 2024), I do not think the convenience of an in-openstack-namespace branch is worth the tradeoffs I have mentioned above.
While I agree with all what Jeremy told me on IRC, and I don't think we should attempt to maintain the CI for too long (as this takes too much of our time), I still believe we should keep a way to share patches. Downstream (like myself for Debian) run their own CI based on packages, and it's always better if we can share patches. It doesn't have to be with a maintained CI. Just having opened branches would be enough.
There's no need for official OpenStack project cooperation to do that. You could store patches in a repo hosted elsewhere, or even have a forked-version of that branch which can be used for a Debian upstream (and other distributors who may be invested in ancient OpenStack versions). I agree cooperation is valuable, but contributor time is valuable and is short supply as well. I-personally am not motivated to spend that limited time giving users an additional excuse to delay their upgrade. -JayF P.S. I fear this came off more harshly than intended; I appreciate the work folks do in projects like Debian to give people a stable base to build on; I just do think that approach is not one we should extend further than we already do as an OpenStack upstream.
On Aug 21, 2025 18:12, Jay Faulkner <jay@gr-oss.io> wrote:
You could store patches in a repo hosted elsewhere
Please name that somewhere. Are you suggesting I maintain my own deployment of Gerrit? or even have a
forked-version of that branch which can be used for a Debian upstream
Well, I use git for packaging, so in some ways, I am doing this already. Though I don't feel like the Debian gitlab instance is a relevant place for this.
I-personally am not motivated to
spend that limited time giving users an additional excuse to delay their
upgrade.
I don't think keeping branches open for *downstream* to work on is giving any message to anyone. Also, the fact users cannot invest enough of their time upgrading is still something unadressed in OpenStack. If we're lacking workforce, there are ways to fix it, like having a release per year instead of 2, so we'd have 3 years of support instead of just one and a half. I know this has been discussed ad-nauseam, but it doesn't remove the concern. And since upstream OpenStack project still hasn't addressed it, what I am asking for is an acceptable middle ground. Cheers, Thomas Goirand P.S: you do realise Red Hat still maintains Train, right ?
On 2025-08-21 19:20:53 +0200 (+0200), thomas@goirand.fr wrote: [...]
Also, the fact users cannot invest enough of their time upgrading is still something unadressed in OpenStack. If we're lacking workforce, there are ways to fix it, like having a release per year instead of 2, so we'd have 3 years of support instead of just one and a half. I know this has been discussed ad-nauseam, but it doesn't remove the concern. And since upstream OpenStack project still hasn't addressed it, what I am asking for is an acceptable middle ground. [...]
I know it doesn't help in the unmaintained/zed case, but starting with the 2023.1 release only one stable branch per year will transition from maintained to unmaintained status (the other will go straight to EOL), and we provide tested guarantees for upgrading between them. This halves the number of unmaintained branches per year going forward, reducing the burden on the community and making it possible to keep more of them open for longer, hopefully. -- Jeremy Stanley
On 21/08/2025 18:41, Jeremy Stanley wrote:
On 2025-08-21 19:20:53 +0200 (+0200), thomas@goirand.fr wrote: [...]
Also, the fact users cannot invest enough of their time upgrading is still something unadressed in OpenStack. If we're lacking workforce, there are ways to fix it, like having a release per year instead of 2, so we'd have 3 years of support instead of just one and a half. I know this has been discussed ad-nauseam, but it doesn't remove the concern. And since upstream OpenStack project still hasn't addressed it, what I am asking for is an acceptable middle ground. [...]
I know it doesn't help in the unmaintained/zed case, but starting with the 2023.1 release only one stable branch per year will transition from maintained to unmaintained status (the other will go straight to EOL), and we provide tested guarantees for upgrading between them. This halves the number of unmaintained branches per year going forward, reducing the burden on the community and making it possible to keep more of them open for longer, hopefully.
we have already EOL'd bobcat in many cases under that policy already. i also don't think we should keep branches open without testing but i was also rignaly against keeping unmaintained branches alive in general. if folks do maintian them with passing tests that one thing but at the very minium they shoudl have tox tests passsing. weather they run tempest is an other story i think they should but unmaintained branches are allowed and encouraged to reduce testing if it is broken and not fixed promptly. https://github.com/openstack/governance/blob/be482953d3dd650ca3128fd0b445664... i think we shoudl be removing all unmainted branche before 2023.1 however since folks hae been maintianing some of the older nova branches i have not really puhsed for that whiel that maintaince continues. https://opendev.org/openstack/nova/branches we can see fixes form elod as recent as 2 months ago for victoria keeping the branches passing in ci https://opendev.org/openstack/nova/commit/de854c89825ce2146f8c1e2a70fe455f78... for watcher there was 0 mantiance happening so we cleaned up the brnach to singlne that nooen shoudl use the older branches https://opendev.org/openstack/watcher/branches out side of a limited set of project most have not had any maintenance on the unmaintained branch in > 6 months https://opendev.org/openstack/cinder/branches https://opendev.org/openstack/glance/branches https://opendev.org/openstack/keystone/branches users of openstack should rightly take that as a signal that they should not be running those older branches from upstream source in production unless they are actively backpacking patches internally. it would be nice to think that if they did that they would do it upstream for all to benefit but that just does not happen. part of the reason we reneamed them to unmaintiend was to signal that there was no commitment or expcation form the core teams to propsose review or merge changes on teh unmaintained branches https://github.com/openstack/governance/blob/master/resolutions/20230724-unm... if other want to collaber on the unmainted brances i dont object ot that but the "The CI for all branches must be in good standing at the time of opt-in." as is """ By default, only the latest eligible Unmaintained branch is kept. When a new branch is eligible, the Unmaintained branch liaison must opt-in to keep all previous branches active. """ https://github.com/openstack/governance/blob/master/resolutions/20230724-unm... in other words ever time we do a release to opt into keeping the older unmainted brances the ci must be in good standing. That is why i think any older unmaintiened branch that is not passing ci shoudl either have it reduced but not elimited or fixed failure ot do that shoudl result in them being EOL'd per the unmaintiend branch policy.
On 8/21/25 10:20 AM, thomas@goirand.fr wrote:
Please name that somewhere. Are you suggesting I maintain my own deployment of Gerrit?
Also, the fact users cannot invest enough of their time upgrading is still something unadressed in OpenStack. If we're lacking workforce,
Create a custom repo using a host of your choice; I would probably use something like codeberg these days. I do not know if the OpenDev team would be willing to host anything unofficial but generally speaking the point is just that: it wouldn't be official and it wouldn't be a sap on the resources of the larger management team. there are ways to fix it, like having a release per year instead of 2, so we'd have 3 years of support instead of just one and a half. As Jeremy already indicated, we have taken these actions including: - reducing the number of branches that extend into stable support which lowers the overall contributor load. I do not wish to reintroduce additional load by adding additional branches to manage. - implementing SLURP so that users have to upgrade half as often as they used to
P.S: you do realise Red Hat still maintains Train, right ?
No, I didn't actually -- and that's the point. Train was released in 2019 -- it's 6 years old at this point. Nobody should run it. If we had a publicly open train branch, even for downstream collaboration, I suspect we'd still be seeing a huge number of operators using train -- even though it doesn't officially support any upstream-supported python versions. If Redhat chooses to support this for their customers, I appreciate that they don't publicize this fact loudly for the reasons I stated in my last email. -JayF
---- On Thu, 21 Aug 2025 08:05:04 -0700 Thomas Goirand <zigo@debian.org> wrote ---
Hi,
It's our policy that we maintain only a few branches, currently, from Caracal to Epoxy (so, 3 stable branches).
Currently, when a Stable branch becomes unsuported, we rename that branch as:
unmaintained/2023.1
which is fine, as we can still commit there. Though after some times, the branches are deleted, and instead, we just do a tag such as:
zed-EOL
Jeremy Stanley just wrote to me on IRC:
"workflows, processes and policies are built around many years of an assumption that unused branches will be deleted"
Keeping unmaintained branches "has put a lot of additional strain on project maintainers and our systems" (still quoting Jeremy).
Though each time there's a new (grave?) security problem, this topic comes back, as we still have no solution for these cases. As a downstream, I still need to write backports for these important patches, sometimes long after the branches are EOL. Red Hat too...
While I agree with all what Jeremy told me on IRC, and I don't think we should attempt to maintain the CI for too long (as this takes too much of our time), I still believe we should keep a way to share patches. Downstream (like myself for Debian) run their own CI based on packages, and it's always better if we can share patches. It doesn't have to be with a maintained CI. Just having opened branches would be enough.
I am so against keeping unmaintained branches for a long time, and even our policy allows that. It needs to be explicitly opt-in by the unmaintained team maintainers, and as you might know, project upstream teams (maintaining master and supported stable branches) are not responsible for maintaining or opt-in the unmaintained branches. For the CI scope, I am very against keeping anything with testing running. It does not make sense to me to have any branch without testing, and we keep backport the changes which can cause a lot of regression. For unmaintained branches, our policy is to at least keep one tempest job testing the overall stack, I do not think that is much work to keep it green if unmaintained branch is really maintained and has volunteers. -gmaan
Your thoughts?
Cheers,
Thomas Goirand (zigo)
participants (6)
-
Ghanshyam Maan
-
Jay Faulkner
-
Jeremy Stanley
-
Sean Mooney
-
Thomas Goirand
-
thomas＠goirand.fr