[tc][all][sahara] Retiring the Sahara project
Hello everyone, As you might know, the Sahara project is not active and was marked 'inactive' since 2024.1 cycle[1]. In this cycle also, this project is leaderless[2] and no maintainer stepped up to maintain it. As discussed in the Leaderless project etherpad[2], the next step is to retire the project. I'm sending this email to propose the Sahara retirement and based on the response, I will proceed next. [1] https://governance.openstack.org/tc/reference/emerging-technology-and-inacti... [2] https://etherpad.opendev.org/p/2024.2-leaderless#L72 -gmann
---- On Wed, 24 Apr 2024 13:39:40 -0700 Ghanshyam Mann wrote ---
Hello everyone,
As you might know, the Sahara project is not active and was marked 'inactive' since 2024.1 cycle[1]. In this cycle also, this project is leaderless[2] and no maintainer stepped up to maintain it. As discussed in the Leaderless project etherpad[2], the next step is to retire the project.
I'm sending this email to propose the Sahara retirement and based on the response, I will proceed next.
Seeing no response to this email or no volunteer to maintain this project, I have proposed the retirement in governance: - https://review.opendev.org/c/openstack/governance/+/919374 -gmann
[1] https://governance.openstack.org/tc/reference/emerging-technology-and-inacti... [2] https://etherpad.opendev.org/p/2024.2-leaderless#L72
-gmann
Just for projects to be aware - since these are now removed from Zuul project list, all projects who still had them in their required-projects for Zuul jobs ended up with broken Zuul config. What's worse, it was failing only *some* jobs that were requiring these repos. As a result, for OpenStack-Ansible around 20 patches landed without gating, as docs job was the only passing one, it was sufficient to set Verified+2. сб, 11 мая 2024 г. в 03:49, Ghanshyam Mann <gmann@ghanshyammann.com>:
---- On Wed, 24 Apr 2024 13:39:40 -0700 Ghanshyam Mann wrote ---
Hello everyone,
As you might know, the Sahara project is not active and was marked 'inactive' since 2024.1 cycle[1]. In this cycle also, this project is leaderless[2] and no maintainer stepped up to maintain it. As discussed in the Leaderless project etherpad[2], the next step is to retire the project.
I'm sending this email to propose the Sahara retirement and based on the response, I will proceed next.
Seeing no response to this email or no volunteer to maintain this project, I have proposed the retirement in governance:
- https://review.opendev.org/c/openstack/governance/+/919374
-gmann
[1] https://governance.openstack.org/tc/reference/emerging-technology-and-inacti... [2] https://etherpad.opendev.org/p/2024.2-leaderless#L72
-gmann
On 2024-05-17 17:27:12 +0200 (+0200), Dmitriy Rabotyagov wrote:
Just for projects to be aware - since these are now removed from Zuul project list, all projects who still had them in their required-projects for Zuul jobs ended up with broken Zuul config.
What's worse, it was failing only *some* jobs that were requiring these repos. As a result, for OpenStack-Ansible around 20 patches landed without gating, as docs job was the only passing one, it was sufficient to set Verified+2. [...]
Yes, that's expected behavior at least from Zuul's perspective. When those projects were removed from the OpenStack tenant, any job definitions which listed them as required-projects (directly or through inheritance) became invalid and were removed from its in-memory configuration. Zuul does test job configuration changes and will refuse to gate any which break job definitions, but there is no similar testing to prevent removal of projects which would result in broken job definitions. Extending Zuul to optionally keep those jobs in the pipeline but force them to insta-fail instead of removing them might be a reasonable feature request, I haven't thought through the possible implications (it could also just be an easy way to create deadlocks though). Similarly we could maybe work out some way to scan Zuul's loaded configuration for any references to projects being removed and then fail testing on the tenant configuration change that proposes to remove them, though that might lead to those changes just never getting merged in some cases. For now, the best approach is to make it clear when projects are proposed for retirement that any other projects who refer to them in their job configs should remove all references at the earliest opportunity, ideally before actual retirement changes merge. Also, keeping a close eye on Zuul's config errors list immediately after such retirement changes merge and reaching out to the affected teams could help further reduce the window for related risks. -- Jeremy Stanley
---- On Fri, 17 May 2024 08:27:12 -0700 Dmitriy Rabotyagov wrote ---
Just for projects to be aware - since these are now removed from Zuul project list, all projects who still had them in their required-projects for Zuul jobs ended up with broken Zuul config.
What's worse, it was failing only *some* jobs that were requiring these repos. As a result, for OpenStack-Ansible around 20 patches landed without gating, as docs job was the only passing one, it was sufficient to set Verified+2.
yeah, there is step#6[1] where we cleanup all references to avoid the error. I was expecting projects to do that in advance when retirement is proposed for example, I see Heat/puppet doing those before repo content and the project is being removed from infra. Anyway, I am going to continue that cleanup today most probably. Also, I agree to move step#6 before step#5 where we remove the project from infra. That will help to avoid such issues. I will propose the same in p-t-g. [1] https://docs.openstack.org/project-team-guide/repository.html#step-6-remove-... -gmann
сб, 11 мая 2024 г. в 03:49, Ghanshyam Mann gmann@ghanshyammann.com>:
---- On Wed, 24 Apr 2024 13:39:40 -0700 Ghanshyam Mann wrote ---
Hello everyone,
As you might know, the Sahara project is not active and was marked 'inactive' since 2024.1 cycle[1]. In this cycle also, this project is leaderless[2] and no maintainer stepped up to maintain it. As discussed in the Leaderless project etherpad[2], the next step is to retire the project.
I'm sending this email to propose the Sahara retirement and based on the response, I will proceed next.
Seeing no response to this email or no volunteer to maintain this project, I have proposed the retirement in governance:
- https://review.opendev.org/c/openstack/governance/+/919374
-gmann
[1] https://governance.openstack.org/tc/reference/emerging-technology-and-inacti... [2] https://etherpad.opendev.org/p/2024.2-leaderless#L72
-gmann
---- On Fri, 17 May 2024 10:25:45 -0700 Ghanshyam Mann wrote ---
---- On Fri, 17 May 2024 08:27:12 -0700 Dmitriy Rabotyagov wrote ---
Just for projects to be aware - since these are now removed from Zuul project list, all projects who still had them in their required-projects for Zuul jobs ended up with broken Zuul config.
What's worse, it was failing only *some* jobs that were requiring these repos. As a result, for OpenStack-Ansible around 20 patches landed without gating, as docs job was the only passing one, it was sufficient to set Verified+2.
yeah, there is step#6[1] where we cleanup all references to avoid the error. I was expecting projects to do that in advance when retirement is proposed for example, I see Heat/puppet doing those before repo content and the project is being removed from infra. Anyway, I am going to continue that cleanup today most probably.
Also, I agree to move step#6 before step#5 where we remove the project from infra. That will help to avoid such issues. I will propose the same in p-t-g.
Done, please check https://review.opendev.org/c/openstack/project-team-guide/+/919976 -gmann
[1] https://docs.openstack.org/project-team-guide/repository.html#step-6-remove-...
-gmann
сб, 11 мая 2024 г. в 03:49, Ghanshyam Mann gmann@ghanshyammann.com>:
---- On Wed, 24 Apr 2024 13:39:40 -0700 Ghanshyam Mann wrote ---
Hello everyone,
As you might know, the Sahara project is not active and was marked 'inactive' since 2024.1 cycle[1]. In this cycle also, this project is leaderless[2] and no maintainer stepped up to maintain it. As discussed in the Leaderless project etherpad[2], the next step is to retire the project.
I'm sending this email to propose the Sahara retirement and based on the response, I will proceed next.
Seeing no response to this email or no volunteer to maintain this project, I have proposed the retirement in governance:
- https://review.opendev.org/c/openstack/governance/+/919374
-gmann
[1] https://governance.openstack.org/tc/reference/emerging-technology-and-inacti... [2] https://etherpad.opendev.org/p/2024.2-leaderless#L72
-gmann
---- On Fri, 17 May 2024 09:03:15 -0700 Jeremy Stanley wrote ---
On 2024-05-17 17:27:12 +0200 (+0200), Dmitriy Rabotyagov wrote:
Just for projects to be aware - since these are now removed from Zuul project list, all projects who still had them in their required-projects for Zuul jobs ended up with broken Zuul config.
What's worse, it was failing only *some* jobs that were requiring these repos. As a result, for OpenStack-Ansible around 20 patches landed without gating, as docs job was the only passing one, it was sufficient to set Verified+2. [...]
Yes, that's expected behavior at least from Zuul's perspective. When those projects were removed from the OpenStack tenant, any job definitions which listed them as required-projects (directly or through inheritance) became invalid and were removed from its in-memory configuration. Zuul does test job configuration changes and will refuse to gate any which break job definitions, but there is no similar testing to prevent removal of projects which would result in broken job definitions.
Extending Zuul to optionally keep those jobs in the pipeline but force them to insta-fail instead of removing them might be a reasonable feature request, I haven't thought through the possible implications (it could also just be an easy way to create deadlocks though). Similarly we could maybe work out some way to scan Zuul's loaded configuration for any references to projects being removed and then fail testing on the tenant configuration change that proposes to remove them, though that might lead to those changes just never getting merged in some cases.
IMO, this should be fixed. ignoring the jobs for any reason which are supposed to run is a false positive. More failure in many cases where it can ignore the things is still better. Either Zuul can run and fail the job or try to ignore the non-existing things (required-projects) and see if the job can pass or not.
For now, the best approach is to make it clear when projects are proposed for retirement that any other projects who refer to them in their job configs should remove all references at the earliest opportunity, ideally before actual retirement changes merge. Also, keeping a close eye on Zuul's config errors list immediately after such retirement changes merge and reaching out to the affected teams could help further reduce the window for related risks.
yeah, I swapped these steps to avoid this but still possible that cleanup changes might merge late than infra removal. https://review.opendev.org/c/openstack/project-team-guide/+/919976 -gmann
-- Jeremy Stanley
Let me a bit iterate of why exactly I feel that even more confusing. OSA has trailing release model, which means our deadline for Caracal is 9th of June. And we did marked corresponding projects as inactive - removed bits for future release [1]. Apparently, removing repos from zuul required projects is not part of marking them as inactive. And retirement notice was for Dalmatian, which I marked in Todo right after releasing Caracal (which has not happened yet). And obviously, everyone else, who's not following trailing release model, already deep in dalmatian. So it also might be possible to do retirement right after ML2, where deadline for trailing is required, to avoid extra confusion on the future. [1] https://opendev.org/openstack/openstack-ansible/commit/930f799141899b0535bdc... On Fri, May 17, 2024, 19:51 Ghanshyam Mann <gmann@ghanshyammann.com> wrote:
---- On Fri, 17 May 2024 09:03:15 -0700 Jeremy Stanley wrote ---
On 2024-05-17 17:27:12 +0200 (+0200), Dmitriy Rabotyagov wrote:
Just for projects to be aware - since these are now removed from Zuul project list, all projects who still had them in their required-projects for Zuul jobs ended up with broken Zuul config.
What's worse, it was failing only *some* jobs that were requiring these repos. As a result, for OpenStack-Ansible around 20 patches landed without gating, as docs job was the only passing one, it was sufficient to set Verified+2. [...]
Yes, that's expected behavior at least from Zuul's perspective. When those projects were removed from the OpenStack tenant, any job definitions which listed them as required-projects (directly or through inheritance) became invalid and were removed from its in-memory configuration. Zuul does test job configuration changes and will refuse to gate any which break job definitions, but there is no similar testing to prevent removal of projects which would result in broken job definitions.
Extending Zuul to optionally keep those jobs in the pipeline but force them to insta-fail instead of removing them might be a reasonable feature request, I haven't thought through the possible implications (it could also just be an easy way to create deadlocks though). Similarly we could maybe work out some way to scan Zuul's loaded configuration for any references to projects being removed and then fail testing on the tenant configuration change that proposes to remove them, though that might lead to those changes just never getting merged in some cases.
IMO, this should be fixed. ignoring the jobs for any reason which are supposed to run is a false positive. More failure in many cases where it can ignore the things is still better. Either Zuul can run and fail the job or try to ignore the non-existing things (required-projects) and see if the job can pass or not.
For now, the best approach is to make it clear when projects are proposed for retirement that any other projects who refer to them in their job configs should remove all references at the earliest opportunity, ideally before actual retirement changes merge. Also, keeping a close eye on Zuul's config errors list immediately after such retirement changes merge and reaching out to the affected teams could help further reduce the window for related risks.
yeah, I swapped these steps to avoid this but still possible that cleanup changes might merge late than infra removal.
https://review.opendev.org/c/openstack/project-team-guide/+/919976
-gmann
-- Jeremy Stanley
---- On Fri, 17 May 2024 11:35:22 -0700 Dmitriy Rabotyagov wrote ---
Let me a bit iterate of why exactly I feel that even more confusing. OSA has trailing release model, which means our deadline for Caracal is 9th of June. And we did marked corresponding projects as inactive - removed bits for future release [1]. Apparently, removing repos from zuul required projects is not part of marking them as inactive. And retirement notice was for Dalmatian, which I marked in Todo right after releasing Caracal (which has not happened yet). And obviously, everyone else, who's not following trailing release model, already deep in dalmatian. So it also might be possible to do retirement right after ML2, where deadline for trailing is required, to avoid extra confusion on the future.
But if you see retirement means stopping the development of the master as well as all the stable branches. It is not related to releases. I know be best is to keep supporting the stable branches but in retirement case where we do not have any maintainers to take care of stable branches, we do not have any other option. Irrespective of Caracal release is done or not, any projects using those retired projects need to do the same step. For example, if project x has a dependency on Sahara then either they need to declare that the Sahara-dependent feature is retired or mention that any issue in Sahara-dependent features cannot be fixed in stable releases (means reply on already used/released Sahara version). -gmann
[1] https://opendev.org/openstack/openstack-ansible/commit/930f799141899b0535bdc... On Fri, May 17, 2024, 19:51 Ghanshyam Mann gmann@ghanshyammann.com> wrote: ---- On Fri, 17 May 2024 09:03:15 -0700 Jeremy Stanley wrote --- > On 2024-05-17 17:27:12 +0200 (+0200), Dmitriy Rabotyagov wrote: > > Just for projects to be aware - since these are now removed from Zuul > > project list, all projects who still had them in their > > required-projects for Zuul jobs ended up with broken Zuul config. > > > > What's worse, it was failing only *some* jobs that were requiring > > these repos. As a result, for OpenStack-Ansible around 20 patches > > landed without gating, as docs job was the only passing one, it was > > sufficient to set Verified+2. > [...] > > Yes, that's expected behavior at least from Zuul's perspective. When > those projects were removed from the OpenStack tenant, any job > definitions which listed them as required-projects (directly or > through inheritance) became invalid and were removed from its > in-memory configuration. Zuul does test job configuration changes > and will refuse to gate any which break job definitions, but there > is no similar testing to prevent removal of projects which would > result in broken job definitions. > > Extending Zuul to optionally keep those jobs in the pipeline but > force them to insta-fail instead of removing them might be a > reasonable feature request, I haven't thought through the possible > implications (it could also just be an easy way to create > deadlocks though). Similarly we could maybe work out some way to > scan Zuul's loaded configuration for any references to projects > being removed and then fail testing on the tenant configuration > change that proposes to remove them, though that might lead to those > changes just never getting merged in some cases.
IMO, this should be fixed. ignoring the jobs for any reason which are supposed to run is a false positive. More failure in many cases where it can ignore the things is still better. Either Zuul can run and fail the job or try to ignore the non-existing things (required-projects) and see if the job can pass or not.
> > For now, the best approach is to make it clear when projects are > proposed for retirement that any other projects who refer to them in > their job configs should remove all references at the earliest > opportunity, ideally before actual retirement changes merge. Also, > keeping a close eye on Zuul's config errors list immediately after > such retirement changes merge and reaching out to the affected teams > could help further reduce the window for related risks.
yeah, I swapped these steps to avoid this but still possible that cleanup changes might merge late than infra removal.
https://review.opendev.org/c/openstack/project-team-guide/+/919976
-gmann
> -- > Jeremy Stanley >
Yeah, that's fair. I guess mixed retirement with deprecation again when was writing previous reply :( On Tue, May 21, 2024, 19:28 Ghanshyam Mann <gmann@ghanshyammann.com> wrote:
---- On Fri, 17 May 2024 11:35:22 -0700 Dmitriy Rabotyagov wrote ---
Let me a bit iterate of why exactly I feel that even more confusing. OSA has trailing release model, which means our deadline for Caracal is 9th of June. And we did marked corresponding projects as inactive - removed bits for future release [1]. Apparently, removing repos from zuul required projects is not part of marking them as inactive. And retirement notice was for Dalmatian, which I marked in Todo right after releasing Caracal (which has not happened yet). And obviously, everyone else, who's not following trailing release model, already deep in dalmatian. So it also might be possible to do retirement right after ML2, where deadline for trailing is required, to avoid extra confusion on the future.
But if you see retirement means stopping the development of the master as well as all the stable branches. It is not related to releases. I know be best is to keep supporting the stable branches but in retirement case where we do not have any maintainers to take care of stable branches, we do not have any other option.
Irrespective of Caracal release is done or not, any projects using those retired projects need to do the same step. For example, if project x has a dependency on Sahara then either they need to declare that the Sahara-dependent feature is retired or mention that any issue in Sahara-dependent features cannot be fixed in stable releases (means reply on already used/released Sahara version).
-gmann
[1]
https://opendev.org/openstack/openstack-ansible/commit/930f799141899b0535bdc...
On Fri, May 17, 2024, 19:51 Ghanshyam Mann gmann@ghanshyammann.com> wrote: ---- On Fri, 17 May 2024 09:03:15 -0700 Jeremy Stanley wrote ---
On 2024-05-17 17:27:12 +0200 (+0200), Dmitriy Rabotyagov wrote:
Just for projects to be aware - since these are now removed from Zuul project list, all projects who still had them in their required-projects for Zuul jobs ended up with broken Zuul config.
What's worse, it was failing only *some* jobs that were requiring these repos. As a result, for OpenStack-Ansible around 20 patches landed without gating, as docs job was the only passing one, it was sufficient to set Verified+2. [...]
Yes, that's expected behavior at least from Zuul's perspective. When those projects were removed from the OpenStack tenant, any job definitions which listed them as required-projects (directly or through inheritance) became invalid and were removed from its in-memory configuration. Zuul does test job configuration changes and will refuse to gate any which break job definitions, but there is no similar testing to prevent removal of projects which would result in broken job definitions.
Extending Zuul to optionally keep those jobs in the pipeline but force them to insta-fail instead of removing them might be a reasonable feature request, I haven't thought through the possible implications (it could also just be an easy way to create deadlocks though). Similarly we could maybe work out some way to scan Zuul's loaded configuration for any references to projects being removed and then fail testing on the tenant configuration change that proposes to remove them, though that might lead to those changes just never getting merged in some cases.
IMO, this should be fixed. ignoring the jobs for any reason which are supposed to run is a false positive. More failure in many cases where it can ignore the things is still better. Either Zuul can run and fail the job or try to ignore the non-existing things (required-projects) and see if the job can pass or not.
For now, the best approach is to make it clear when projects are proposed for retirement that any other projects who refer to them in their job configs should remove all references at the earliest opportunity, ideally before actual retirement changes merge. Also, keeping a close eye on Zuul's config errors list immediately after such retirement changes merge and reaching out to the affected teams could help further reduce the window for related risks.
yeah, I swapped these steps to avoid this but still possible that cleanup changes might merge late than infra removal.
https://review.opendev.org/c/openstack/project-team-guide/+/919976
-gmann
-- Jeremy Stanley
participants (3)
-
Dmitriy Rabotyagov
-
Ghanshyam Mann
-
Jeremy Stanley