Hello OpenStack community
I'm one of the members of the StarlingX community. We've had a lot of stability issues with our ability to compile both our development branch and supported release branches these last few weeks. It all traces back to the Train EOL. We weren't monitoring openstack mailing lists, and missed the EOL announcement. We are actively moving off of Train, but aren't yet ready.
What's really causing us grief is that some sub-projects, e.g.heat and nova, have started deleting elements of Train, e.g. git branches.
Now please don't take this wrong. Ending support for an old branch is a totally normal thing, and we accept that. If StarlingX customers need support in that area, we'll provide it. However I would plea to you to NOT delete the elements of Train that allow other projects to compile old openstack releases, e.g. your gits branches.
Sincerely
Scott Little on behalf of StarlingX
Hey Scott,
Good news! It's all still there, just not as branches. When a branch is moved from "Extended Maintenance" (EM) to End of Life (EOL), we remove the branch but retain the git refs on a tag.[1] (e.g. https://opendev.org/openstack/ironic/src/tag/stein-eol is the tag representing Ironic stable/stein at the point of EOL).
Look for the `train-eol` tag on the projects you're struggling with, and that should be the git ref you're looking for. Hopefully your tooling is happy getting any git ref and not just a branch ref.
1: https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-li...
Thanks, Jay Faulkner Ironic PTL TC Vice-Chair
On Thu, May 11, 2023 at 3:20 PM Scott Little scott.little@windriver.com wrote:
Hello OpenStack community
I'm one of the members of the StarlingX community. We've had a lot of stability issues with our ability to compile both our development branch and supported release branches these last few weeks. It all traces back to the Train EOL. We weren't monitoring openstack mailing lists, and missed the EOL announcement. We are actively moving off of Train, but aren't yet ready.
What's really causing us grief is that some sub-projects, e.g.heat and nova, have started deleting elements of Train, e.g. git branches.
Now please don't take this wrong. Ending support for an old branch is a totally normal thing, and we accept that. If StarlingX customers need support in that area, we'll provide it. However I would plea to you to NOT delete the elements of Train that allow other projects to compile old openstack releases, e.g. your gits branches.
Sincerely
Scott Little on behalf of StarlingX
Thanks for your response Jay
Yes the <branch>-eol tag is somewhat useful, but it doesn't appear to be created until the branch is removed. There is no way for a downstream consumer to to prepare for the forthcoming branch deletion. There is no way to avoid a period of breakage.
I'd like to suggest the branches be 'frozen', as in accepting no further updates, and tagged with <branch>-eol, but otherwise allowed to continue to exist for several months if not a year. That would allow downstream projects a reasonable period to switch from branch to tag and avoid a period of breakage.
Second point. Is there a separate mailing list to announce major events such as EOL of a branch? It's hard to pick such announcements out of a general mailing list.
Scott
On 2023-05-11 18:39, Jay Faulkner wrote:
** *CAUTION: This email comes from a non Wind River email account!* Do not click links or open attachments unless you recognize the sender and know the content is safe. Hey Scott,
Good news! It's all still there, just not as branches. When a branch is moved from "Extended Maintenance" (EM) to End of Life (EOL), we remove the branch but retain the git refs on a tag.[1] (e.g. https://opendev.org/openstack/ironic/src/tag/stein-eol https://urldefense.com/v3/__https://opendev.org/openstack/ironic/src/tag/stein-eol__;!!AjveYdw8EvQ!apBHo-KzBygqt5lBcWQglBwJIGOdfH3sz3CcLWJ28X_9PFacGxLhRQnyyxD8ARoFb-md29v17QlQmDvFpQ$ is the tag representing Ironic stable/stein at the point of EOL).
Look for the `train-eol` tag on the projects you're struggling with, and that should be the git ref you're looking for. Hopefully your tooling is happy getting any git ref and not just a branch ref.
1: https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-li... https://urldefense.com/v3/__https://docs.openstack.org/project-team-guide/stable-branches.html*end-of-life__;Iw!!AjveYdw8EvQ!apBHo-KzBygqt5lBcWQglBwJIGOdfH3sz3CcLWJ28X_9PFacGxLhRQnyyxD8ARoFb-md29v17QlQZtReEw$
Thanks, Jay Faulkner Ironic PTL TC Vice-Chair
On Thu, May 11, 2023 at 3:20 PM Scott Little scott.little@windriver.com wrote:
Hello OpenStack community I'm one of the members of the StarlingX community. We've had a lot of stability issues with our ability to compile both our development branch and supported release branches these last few weeks. It all traces back to the Train EOL. We weren't monitoring openstack mailing lists, and missed the EOL announcement. We are actively moving off of Train, but aren't yet ready. What's really causing us grief is that some sub-projects, e.g.heat and nova, have started deleting elements of Train, e.g. git branches. Now please don't take this wrong. Ending support for an old branch is a totally normal thing, and we accept that. If StarlingX customers need support in that area, we'll provide it. However I would plea to you to NOT delete the elements of Train that allow other projects to compile old openstack releases, e.g. your gits branches. Sincerely Scott Little on behalf of StarlingX
On 2023-05-12 09:32:19 -0400 (-0400), Scott Little wrote: [...]
I'd like to suggest the branches be 'frozen', as in accepting no further updates, and tagged with <branch>-eol, but otherwise allowed to continue to exist for several months if not a year. That would allow downstream projects a reasonable period to switch from branch to tag and avoid a period of breakage.
We can't easily make changes proposed for those branches get automatically rejected without deleting the branches, but also the deletion is meant to send a clear and noticeable signal to anyone still trying to pull from it, while an EOL tag may go unnoticed.
The branch was effectively frozen (as in no new stable point releases were made) the moment normal maintenance of that branch ended, and then several years went by where users were expected to stop relying on it. The EOL tagging and subsequent branch cleanup is the final signal to anyone why may have not otherwise noticed we stopped maintaining and releasing the branch years prior.
We've had some discussions about the term "extended maintenance" being confusing to consumers who think that means the branch is still maintained and recommended for direct use. I've been advocating we refer to everything after the end of normal maintenance as "unmaintained" in order to make that more clear, though there was another suggestion for "community maintenance" which might be closer to the truth for some projects.
Second point. Is there a separate mailing list to announce major events such as EOL of a branch? It's hard to pick such announcements out of a general mailing list.
I can see making a case for announcing the end of normal maintenance on the openstack-announce mailing list. EOL of individual projects on the other hand is going to be too high-volume for an announcements since they are not required to coordinate that with each other and can do so any time between the end of normal maintenance and the mandatory sunset.
Scott,
I acknowledge and empathize with the pain -- I've operated OpenStack forks downstream at previous jobs and have had to push changes to swap from the branch to the EOL-tag. With that being said, I still wouldn't be in favor of that kind of change in policy, because the downsides -- such as it not being clear to contributors what branches were open for contribution -- are pretty high.
We have an announcement list[1]. I don't think it's a bad idea to start publishing posts there when a branch changes support status. I'm curious what other contributors would think of this.
Thanks, Jay Faulkner Ironic PTL TC Vice-Chair
1: https://lists.openstack.org/pipermail/openstack-announce/
On Fri, May 12, 2023 at 6:32 AM Scott Little scott.little@windriver.com wrote:
Thanks for your response Jay
Yes the <branch>-eol tag is somewhat useful, but it doesn't appear to be created until the branch is removed. There is no way for a downstream consumer to to prepare for the forthcoming branch deletion. There is no way to avoid a period of breakage.
I'd like to suggest the branches be 'frozen', as in accepting no further updates, and tagged with <branch>-eol, but otherwise allowed to continue to exist for several months if not a year. That would allow downstream projects a reasonable period to switch from branch to tag and avoid a period of breakage.
Second point. Is there a separate mailing list to announce major events such as EOL of a branch? It's hard to pick such announcements out of a general mailing list.
Scott
On 2023-05-11 18:39, Jay Faulkner wrote:
*CAUTION: This email comes from a non Wind River email account!* Do not click links or open attachments unless you recognize the sender and know the content is safe. Hey Scott,
Good news! It's all still there, just not as branches. When a branch is moved from "Extended Maintenance" (EM) to End of Life (EOL), we remove the branch but retain the git refs on a tag.[1] (e.g. https://opendev.org/openstack/ironic/src/tag/stein-eol https://urldefense.com/v3/__https://opendev.org/openstack/ironic/src/tag/stein-eol__;!!AjveYdw8EvQ!apBHo-KzBygqt5lBcWQglBwJIGOdfH3sz3CcLWJ28X_9PFacGxLhRQnyyxD8ARoFb-md29v17QlQmDvFpQ$ is the tag representing Ironic stable/stein at the point of EOL).
Look for the `train-eol` tag on the projects you're struggling with, and that should be the git ref you're looking for. Hopefully your tooling is happy getting any git ref and not just a branch ref.
1: https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-li... https://urldefense.com/v3/__https://docs.openstack.org/project-team-guide/stable-branches.html*end-of-life__;Iw!!AjveYdw8EvQ!apBHo-KzBygqt5lBcWQglBwJIGOdfH3sz3CcLWJ28X_9PFacGxLhRQnyyxD8ARoFb-md29v17QlQZtReEw$
Thanks, Jay Faulkner Ironic PTL TC Vice-Chair
On Thu, May 11, 2023 at 3:20 PM Scott Little scott.little@windriver.com wrote:
Hello OpenStack community
I'm one of the members of the StarlingX community. We've had a lot of stability issues with our ability to compile both our development branch and supported release branches these last few weeks. It all traces back to the Train EOL. We weren't monitoring openstack mailing lists, and missed the EOL announcement. We are actively moving off of Train, but aren't yet ready.
What's really causing us grief is that some sub-projects, e.g.heat and nova, have started deleting elements of Train, e.g. git branches.
Now please don't take this wrong. Ending support for an old branch is a totally normal thing, and we accept that. If StarlingX customers need support in that area, we'll provide it. However I would plea to you to NOT delete the elements of Train that allow other projects to compile old openstack releases, e.g. your gits branches.
Sincerely
Scott Little on behalf of StarlingX
Thanks Jay & Jeremy for all the answers you gave to Scott. Let me also emphasize (with my stable maintainer hat on 🙂) that Train is *not* End of Life yet [1], but some of the projects decided to remove some of their stable/train (and even newer) branches due to lack of maintainers and broken gates and to free up resources.
(Note that, for example core projects have EOL'd their stable/rocky branches for quite a while now, and Rocky is about to EOL as a whole. Stein is in the very same situation (broken gates, long gone core components, etc.) and it is inevitable that Train will reach that point, too.)
Nevertheless, we also discussed recently that perhaps it would be beneficial to keep old branches open instead of deleting them, but for different reasons teams were not favor of doing that. Jay had a very good point in his previous mail as an example why it is better to delete old branches.
[1] https://releases.openstack.org/
Thanks, Előd Illés irc: elodilles @ #openstack-stable #openstack-release
________________________________ From: Jay Faulkner jay@gr-oss.io Sent: Friday, May 12, 2023 4:50 PM To: Scott Little scott.little@windriver.com Cc: openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org Subject: Re: Train EOL
Scott,
I acknowledge and empathize with the pain -- I've operated OpenStack forks downstream at previous jobs and have had to push changes to swap from the branch to the EOL-tag. With that being said, I still wouldn't be in favor of that kind of change in policy, because the downsides -- such as it not being clear to contributors what branches were open for contribution -- are pretty high.
We have an announcement list[1]. I don't think it's a bad idea to start publishing posts there when a branch changes support status. I'm curious what other contributors would think of this.
Thanks, Jay Faulkner Ironic PTL TC Vice-Chair
1: https://lists.openstack.org/pipermail/openstack-announce/
On Fri, May 12, 2023 at 6:32 AM Scott Little <scott.little@windriver.commailto:scott.little@windriver.com> wrote:
Thanks for your response Jay
Yes the <branch>-eol tag is somewhat useful, but it doesn't appear to be created until the branch is removed. There is no way for a downstream consumer to to prepare for the forthcoming branch deletion. There is no way to avoid a period of breakage.
I'd like to suggest the branches be 'frozen', as in accepting no further updates, and tagged with <branch>-eol, but otherwise allowed to continue to exist for several months if not a year. That would allow downstream projects a reasonable period to switch from branch to tag and avoid a period of breakage.
Second point. Is there a separate mailing list to announce major events such as EOL of a branch? It's hard to pick such announcements out of a general mailing list.
Scott
On 2023-05-11 18:39, Jay Faulkner wrote: CAUTION: This email comes from a non Wind River email account! Do not click links or open attachments unless you recognize the sender and know the content is safe. Hey Scott,
Good news! It's all still there, just not as branches. When a branch is moved from "Extended Maintenance" (EM) to End of Life (EOL), we remove the branch but retain the git refs on a tag.[1] (e.g. https://opendev.org/openstack/ironic/src/tag/stein-eolhttps://urldefense.com/v3/__https://opendev.org/openstack/ironic/src/tag/stein-eol__;!!AjveYdw8EvQ!apBHo-KzBygqt5lBcWQglBwJIGOdfH3sz3CcLWJ28X_9PFacGxLhRQnyyxD8ARoFb-md29v17QlQmDvFpQ$ is the tag representing Ironic stable/stein at the point of EOL).
Look for the `train-eol` tag on the projects you're struggling with, and that should be the git ref you're looking for. Hopefully your tooling is happy getting any git ref and not just a branch ref.
1: https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-li...https://urldefense.com/v3/__https://docs.openstack.org/project-team-guide/stable-branches.html*end-of-life__;Iw!!AjveYdw8EvQ!apBHo-KzBygqt5lBcWQglBwJIGOdfH3sz3CcLWJ28X_9PFacGxLhRQnyyxD8ARoFb-md29v17QlQZtReEw$
Thanks, Jay Faulkner Ironic PTL TC Vice-Chair
On Thu, May 11, 2023 at 3:20 PM Scott Little <scott.little@windriver.commailto:scott.little@windriver.com> wrote: Hello OpenStack community
I'm one of the members of the StarlingX community. We've had a lot of stability issues with our ability to compile both our development branch and supported release branches these last few weeks. It all traces back to the Train EOL. We weren't monitoring openstack mailing lists, and missed the EOL announcement. We are actively moving off of Train, but aren't yet ready.
What's really causing us grief is that some sub-projects, e.g.heat and nova, have started deleting elements of Train, e.g. git branches.
Now please don't take this wrong. Ending support for an old branch is a totally normal thing, and we accept that. If StarlingX customers need support in that area, we'll provide it. However I would plea to you to NOT delete the elements of Train that allow other projects to compile old openstack releases, e.g. your gits branches.
Sincerely
Scott Little on behalf of StarlingX
On Sat, 13 May 2023 at 00:35, Scott Little scott.little@windriver.com wrote:
Thanks for your response Jay
Yes the <branch>-eol tag is somewhat useful, but it doesn't appear to be created until the branch is removed. There is no way for a downstream consumer to to prepare for the forthcoming branch deletion. There is no way to avoid a period of breakage.
This doesn't help you for train, but may save you some headaches in the future? You can keep an eye out for <branch>-em tags arriving. They're created when a project moves into Extended Maintenance mode. That's a pretty good signal that downstream consumers should expect the branch to translation to -eol "soon". Soon is hard to define but essentially it's a 6 month warning.
Tony.
On 2023-05-11 16:27:56 -0400 (-0400), Scott Little wrote: [...]
I would plea to you to NOT delete the elements of Train that allow other projects to compile old openstack releases, e.g. your gits branches.
For future reference, it's unsafe to assume branches stick around forever. Projects may EOL and delete them as soon as 18 months from the coordinated release (when normal maintenance for those branches ends). Tags, on the other hand, are kept indefinitely. If you need to know the most recent stable point release tag for a given coordinated release series, you can find them listed on the releases site: As an example, https://releases.openstack.org/train/index.html lists them for Train.
While many projects practice an "extended maintenance" to allow interested members of the community to continue backporting fixes on stable branches after that point, those branches are intended as a point of coordination for downstream maintainers who want to collaborate upstream on developing and reviewing backports for older releases. Branches under extended maintenance don't receive point releases and aren't intended to be consumed directly for deployment, but are rather meant as a source for cherry-picking relevant fixes.
Not all projects practice extended maintenance and may EOL and delete their branches at any point after normal maintenance ends, but even those who do extend maintenance as long as possible are eventually sunset in a similar fashion. Mandatory sunset is currently about 5 years from the initial coordinated release for that series, but we probably need to shorten it given challenges with testing 10 different branches across projects and lack of actual extended maintenance interest from downstream consumers and redistributors (they're interested in things being maintained of course, but not doing the work to make it happen).
participants (5)
-
Elõd Illés
-
Jay Faulkner
-
Jeremy Stanley
-
Scott Little
-
Tony Breeds