[ironic][release] Bugfix branch status and cleanup w/r/t zuul-config-errors
Hey all, I've been looking into the various zuul config errors showing up for Ironic-program branches. Almost all of our old bugfix branches are in the list. Additionally, not properly retiring the bugfix branches leads to an ever-growing list of branches which makes it a lot more difficult, for contributors and operators alike, to tell which ones are currently supported. I've put together a document describing the situation as it is now, and my proposal: https://etherpad.opendev.org/p/IronicBugfixBranchCleanup Essentially, I think we need to: - identify bugfix branches to cleanup (I've done this in the above etherpad, but some of the ) - clean them up (the next step) - update Ironic policy to set a regular cadence for when to retire bugfix branches, and encode the process for doing so This means there are two overall questions to answer in this email: 1) Mechanically, what's the process for doing this? I don't believe the existing release tooling will be useful for this, but I'm not 100% sure. I've pulled (in the above etherpad and a local spreadsheet) the last SHA for each branch; so we should be able to EOL these branches similarly to how we EOL stable branches; except manually instead of with tooling. Who is going to do this work? (I'd prefer releases team continue to hold the keys to do this; but I understand if you don't want to take on this manual work). 2) What's the pattern for Ironic to adopt regarding these branches? We just need to write down the expected lifecycle and enforce it -- so we prevent being this deep into "branch debt" in the future. What do folks think? - Jay Faulkner
Hi Jay, On Tue, Nov 1, 2022 at 8:17 PM Jay Faulkner <jay@gr-oss.io> wrote:
Hey all,
I've been looking into the various zuul config errors showing up for Ironic-program branches. Almost all of our old bugfix branches are in the list. Additionally, not properly retiring the bugfix branches leads to an ever-growing list of branches which makes it a lot more difficult, for contributors and operators alike, to tell which ones are currently supported.
I'd like to see the errors. We update Zuul configuration manually for each bugfix branch, mapping appropriate branches for other projects (devstack, nova, etc). It's possible that we always overlook a few jobs, which causes Zuul to be upset (but quietly upset, so we don't notice).
I've put together a document describing the situation as it is now, and my proposal: https://etherpad.opendev.org/p/IronicBugfixBranchCleanup
Going with the "I would like to retire" would cause us so much trouble that we'll have to urgently create a downstream mirror of them. Once we do this, using upstream bugfix branches at all will be questionable. Especially bugfix/19.0 (and corresponding IPA/inspector branches) is used in a very actively maintained release.
Essentially, I think we need to: - identify bugfix branches to cleanup (I've done this in the above etherpad, but some of the ) - clean them up (the next step) - update Ironic policy to set a regular cadence for when to retire bugfix branches, and encode the process for doing so
This means there are two overall questions to answer in this email: 1) Mechanically, what's the process for doing this? I don't believe the existing release tooling will be useful for this, but I'm not 100% sure. I've pulled (in the above etherpad and a local spreadsheet) the last SHA for each branch; so we should be able to EOL these branches similarly to how we EOL stable branches; except manually instead of with tooling. Who is going to do this work? (I'd prefer releases team continue to hold the keys to do this; but I understand if you don't want to take on this manual work).
EOL tags will be created by the release team, yes. I don't think we can get the keys without going "independent".
2) What's the pattern for Ironic to adopt regarding these branches? We just need to write down the expected lifecycle and enforce it -- so we prevent being this deep into "branch debt" in the future.
With my vendor's (red) hat on, I'd prefer to have a dual approach: the newest branches are supported by the community (i.e. us all), the oldest - by vendors who need them (EOLed if nobody volunteers). I think you already have a list of branches that OCP uses? Feel free to point Riccardo, Iury or myself at any issues with them. Dmitry
What do folks think?
- Jay Faulkner
-- Red Hat GmbH <https://www.redhat.com/de/global/dach>, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill, Amy Ross
On Wed, Nov 2, 2022 at 3:04 AM Dmitry Tantsur <dtantsur@redhat.com> wrote:
Hi Jay,
On Tue, Nov 1, 2022 at 8:17 PM Jay Faulkner <jay@gr-oss.io> wrote:
Hey all,
I've been looking into the various zuul config errors showing up for Ironic-program branches. Almost all of our old bugfix branches are in the list. Additionally, not properly retiring the bugfix branches leads to an ever-growing list of branches which makes it a lot more difficult, for contributors and operators alike, to tell which ones are currently supported.
I'd like to see the errors. We update Zuul configuration manually for each bugfix branch, mapping appropriate branches for other projects (devstack, nova, etc). It's possible that we always overlook a few jobs, which causes Zuul to be upset (but quietly upset, so we don't notice).
The errors show up in https://zuul.opendev.org/t/openstack/config-errors -- although they seem to be broken this morning. Most of them are older bugfix branches, ones that are out of support, that have the `Queue: Ironic` param that's no longer supported. I am not in favor of anyone going to dead bugfix branches and fixing CI; instead we should retire the ones out of use.
I've put together a document describing the situation as it is now, and my proposal: https://etherpad.opendev.org/p/IronicBugfixBranchCleanup
Going with the "I would like to retire" would cause us so much trouble that we'll have to urgently create a downstream mirror of them. Once we do this, using upstream bugfix branches at all will be questionable. Especially bugfix/19.0 (and corresponding IPA/inspector branches) is used in a very actively maintained release.
Then we won't; but we do need to think about what timeline we can talk about upstream for getting a cadence for getting these retired out, just like we have a cadence for getting them cut every two months. I'll revise the list and remove the "I would like to retire" section (move it to keep-em-up).
Essentially, I think we need to: - identify bugfix branches to cleanup (I've done this in the above etherpad, but some of the ) - clean them up (the next step) - update Ironic policy to set a regular cadence for when to retire bugfix branches, and encode the process for doing so
This means there are two overall questions to answer in this email: 1) Mechanically, what's the process for doing this? I don't believe the existing release tooling will be useful for this, but I'm not 100% sure. I've pulled (in the above etherpad and a local spreadsheet) the last SHA for each branch; so we should be able to EOL these branches similarly to how we EOL stable branches; except manually instead of with tooling. Who is going to do this work? (I'd prefer releases team continue to hold the keys to do this; but I understand if you don't want to take on this manual work).
EOL tags will be created by the release team, yes. I don't think we can get the keys without going "independent".
It's a gerrit ACL you can enable to give other people access to tags; but like I said, I don't want that access anyway :).
2) What's the pattern for Ironic to adopt regarding these branches? We just need to write down the expected lifecycle and enforce it -- so we prevent being this deep into "branch debt" in the future.
With my vendor's (red) hat on, I'd prefer to have a dual approach: the newest branches are supported by the community (i.e. us all), the oldest - by vendors who need them (EOLed if nobody volunteers). I think you already have a list of branches that OCP uses? Feel free to point Riccardo, Iury or myself at any issues with them.
That's not really an option IMO. These branches exist in the upstream community, and are seen by upstream contributors and operators. If they're going to live here; they need to have some reasonable documentation about what folks should expect out of them and efforts being put towards them. Even if the documentation is "bugfix/1.2 is maintained as long as Product A 1.2 is maintained", that's better than leaving the community guessing about what these are used for, and why some are more-supported than others. -Jay Dmitry
What do folks think?
- Jay Faulkner
--
Red Hat GmbH <https://www.redhat.com/de/global/dach>, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill, Amy Ross
Does anyone from the Releases team want to chime in on the best way to execute this kind of change? -JayF On Wed, Nov 2, 2022 at 7:02 AM Jay Faulkner <jay@gr-oss.io> wrote:
On Wed, Nov 2, 2022 at 3:04 AM Dmitry Tantsur <dtantsur@redhat.com> wrote:
Hi Jay,
On Tue, Nov 1, 2022 at 8:17 PM Jay Faulkner <jay@gr-oss.io> wrote:
Hey all,
I've been looking into the various zuul config errors showing up for Ironic-program branches. Almost all of our old bugfix branches are in the list. Additionally, not properly retiring the bugfix branches leads to an ever-growing list of branches which makes it a lot more difficult, for contributors and operators alike, to tell which ones are currently supported.
I'd like to see the errors. We update Zuul configuration manually for each bugfix branch, mapping appropriate branches for other projects (devstack, nova, etc). It's possible that we always overlook a few jobs, which causes Zuul to be upset (but quietly upset, so we don't notice).
The errors show up in https://zuul.opendev.org/t/openstack/config-errors -- although they seem to be broken this morning. Most of them are older bugfix branches, ones that are out of support, that have the `Queue: Ironic` param that's no longer supported. I am not in favor of anyone going to dead bugfix branches and fixing CI; instead we should retire the ones out of use.
I've put together a document describing the situation as it is now, and my proposal: https://etherpad.opendev.org/p/IronicBugfixBranchCleanup
Going with the "I would like to retire" would cause us so much trouble that we'll have to urgently create a downstream mirror of them. Once we do this, using upstream bugfix branches at all will be questionable. Especially bugfix/19.0 (and corresponding IPA/inspector branches) is used in a very actively maintained release.
Then we won't; but we do need to think about what timeline we can talk about upstream for getting a cadence for getting these retired out, just like we have a cadence for getting them cut every two months. I'll revise the list and remove the "I would like to retire" section (move it to keep-em-up).
Essentially, I think we need to: - identify bugfix branches to cleanup (I've done this in the above etherpad, but some of the ) - clean them up (the next step) - update Ironic policy to set a regular cadence for when to retire bugfix branches, and encode the process for doing so
This means there are two overall questions to answer in this email: 1) Mechanically, what's the process for doing this? I don't believe the existing release tooling will be useful for this, but I'm not 100% sure. I've pulled (in the above etherpad and a local spreadsheet) the last SHA for each branch; so we should be able to EOL these branches similarly to how we EOL stable branches; except manually instead of with tooling. Who is going to do this work? (I'd prefer releases team continue to hold the keys to do this; but I understand if you don't want to take on this manual work).
EOL tags will be created by the release team, yes. I don't think we can get the keys without going "independent".
It's a gerrit ACL you can enable to give other people access to tags; but like I said, I don't want that access anyway :).
2) What's the pattern for Ironic to adopt regarding these branches? We just need to write down the expected lifecycle and enforce it -- so we prevent being this deep into "branch debt" in the future.
With my vendor's (red) hat on, I'd prefer to have a dual approach: the newest branches are supported by the community (i.e. us all), the oldest - by vendors who need them (EOLed if nobody volunteers). I think you already have a list of branches that OCP uses? Feel free to point Riccardo, Iury or myself at any issues with them.
That's not really an option IMO. These branches exist in the upstream community, and are seen by upstream contributors and operators. If they're going to live here; they need to have some reasonable documentation about what folks should expect out of them and efforts being put towards them. Even if the documentation is "bugfix/1.2 is maintained as long as Product A 1.2 is maintained", that's better than leaving the community guessing about what these are used for, and why some are more-supported than others.
-Jay
Dmitry
What do folks think?
- Jay Faulkner
--
Red Hat GmbH <https://www.redhat.com/de/global/dach>, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill, Amy Ross
Hi, TL;DR: for a quick solution I suggest to add the ACL [1] for the ironic cores, something like this: [access "refs/heads/bugfix/*"] delete = = group ironic-core The more lengthy answer: as far as I know, bugfix branches were allowed based on ironic team's request, but with the conditions, that * it is out of Release Management scope * no official releases will be made out of these branches So basically it's completely the Ironic team's responsibility. (More senior @Release Managers: correct me if i am wrong) Nevertheless, time to time, there seem to be a need for 'EOL'ing / deleting branches that are not in the releases repo's deliverables/* directory (for example: projects opened a stable branch manually; no tags / releases were administered; branches were reopened accidentally; given project was not part of the official series release; etc). These needs to be handled out of the releases repo 'deliverables' directory, thus needs different handling & tools. And this task is not really coupled with release management, and also note, that the release management team is a very small team nowadays. Anyway, as a general stable maintainer core, I'm also thinking about how to solve this issue (old, obsolete branch deletion) in a safe way, though I'm not there yet to have any concrete solution. Ideas are welcome 🙂 . Until a solution is not found, the quickest solution is I think is to add the above ACL extension to the ironic.config [1]. Cheers, Előd Illés irc: elodilles @ #openstack-release #openstack-stable [1] https://opendev.org/openstack/project-config/src/commit/e92af53c10a811f4370c... ________________________________ From: Jay Faulkner <jay@gr-oss.io> Sent: Tuesday, November 8, 2022 11:33 PM To: OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: Re: [ironic][release] Bugfix branch status and cleanup w/r/t zuul-config-errors Does anyone from the Releases team want to chime in on the best way to execute this kind of change? -JayF On Wed, Nov 2, 2022 at 7:02 AM Jay Faulkner <jay@gr-oss.io<mailto:jay@gr-oss.io>> wrote: On Wed, Nov 2, 2022 at 3:04 AM Dmitry Tantsur <dtantsur@redhat.com<mailto:dtantsur@redhat.com>> wrote: Hi Jay, On Tue, Nov 1, 2022 at 8:17 PM Jay Faulkner <jay@gr-oss.io<mailto:jay@gr-oss.io>> wrote: Hey all, I've been looking into the various zuul config errors showing up for Ironic-program branches. Almost all of our old bugfix branches are in the list. Additionally, not properly retiring the bugfix branches leads to an ever-growing list of branches which makes it a lot more difficult, for contributors and operators alike, to tell which ones are currently supported. I'd like to see the errors. We update Zuul configuration manually for each bugfix branch, mapping appropriate branches for other projects (devstack, nova, etc). It's possible that we always overlook a few jobs, which causes Zuul to be upset (but quietly upset, so we don't notice). The errors show up in https://zuul.opendev.org/t/openstack/config-errors -- although they seem to be broken this morning. Most of them are older bugfix branches, ones that are out of support, that have the `Queue: Ironic` param that's no longer supported. I am not in favor of anyone going to dead bugfix branches and fixing CI; instead we should retire the ones out of use. I've put together a document describing the situation as it is now, and my proposal: https://etherpad.opendev.org/p/IronicBugfixBranchCleanup Going with the "I would like to retire" would cause us so much trouble that we'll have to urgently create a downstream mirror of them. Once we do this, using upstream bugfix branches at all will be questionable. Especially bugfix/19.0 (and corresponding IPA/inspector branches) is used in a very actively maintained release. Then we won't; but we do need to think about what timeline we can talk about upstream for getting a cadence for getting these retired out, just like we have a cadence for getting them cut every two months. I'll revise the list and remove the "I would like to retire" section (move it to keep-em-up). Essentially, I think we need to: - identify bugfix branches to cleanup (I've done this in the above etherpad, but some of the ) - clean them up (the next step) - update Ironic policy to set a regular cadence for when to retire bugfix branches, and encode the process for doing so This means there are two overall questions to answer in this email: 1) Mechanically, what's the process for doing this? I don't believe the existing release tooling will be useful for this, but I'm not 100% sure. I've pulled (in the above etherpad and a local spreadsheet) the last SHA for each branch; so we should be able to EOL these branches similarly to how we EOL stable branches; except manually instead of with tooling. Who is going to do this work? (I'd prefer releases team continue to hold the keys to do this; but I understand if you don't want to take on this manual work). EOL tags will be created by the release team, yes. I don't think we can get the keys without going "independent". It's a gerrit ACL you can enable to give other people access to tags; but like I said, I don't want that access anyway :). 2) What's the pattern for Ironic to adopt regarding these branches? We just need to write down the expected lifecycle and enforce it -- so we prevent being this deep into "branch debt" in the future. With my vendor's (red) hat on, I'd prefer to have a dual approach: the newest branches are supported by the community (i.e. us all), the oldest - by vendors who need them (EOLed if nobody volunteers). I think you already have a list of branches that OCP uses? Feel free to point Riccardo, Iury or myself at any issues with them. That's not really an option IMO. These branches exist in the upstream community, and are seen by upstream contributors and operators. If they're going to live here; they need to have some reasonable documentation about what folks should expect out of them and efforts being put towards them. Even if the documentation is "bugfix/1.2 is maintained as long as Product A 1.2 is maintained", that's better than leaving the community guessing about what these are used for, and why some are more-supported than others. -Jay Dmitry What do folks think? - Jay Faulkner -- Red Hat GmbH<https://www.redhat.com/de/global/dach>, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243, Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill, Amy Ross
On 2022-12-01 18:04:48 +0000 (+0000), Elõd Illés wrote:
TL;DR: for a quick solution I suggest to add the ACL [1] for the ironic cores, something like this:
[access "refs/heads/bugfix/*"] delete = = group ironic-core [...]
And if you're worried about some of the core reviewers accidentally deleting branches, you could make that a different group with a smaller set of people (perhaps just the PTL for example). And if you're wanting to be super extra careful you could create a second Ubuntu One account with a different E-mail address and use that to create a new Gerrit account which you only log into in order to perform these sorts of tasks (may be overkill for this case, but it's similar to what our Gerrit admins do in order to not pollute their normal reviewer accounts with admin level permissions). -- Jeremy Stanley
Elod, thanks for the insight. We discussed this at Monday's Ironic meeting and decided to create a small, more limited group to gain this access. I've taken the first step to enable this access via this merge request: https://review.opendev.org/admin/groups/0c53b8f80897aa9e7cee7347e4710bd9b8bd... -- once it's completed, I'll ask gerrit admins to give me access to modify membership. Once that's done, the Ironic cores who will be taking this access are me+our release liaisons, Dmitry, Iury, and Riccardo. Thanks, Jay Faulkner On Thu, Dec 1, 2022 at 10:04 AM Elõd Illés <elod.illes@est.tech> wrote:
Hi,
TL;DR: for a quick solution I suggest to add the ACL [1] for the ironic cores, something like this:
[access "refs/heads/bugfix/*"] delete = = group ironic-core
The more lengthy answer: as far as I know, bugfix branches were allowed based on ironic team's request, but with the conditions, that
- it is out of Release Management scope - no official releases will be made out of these branches
So basically it's completely the Ironic team's responsibility. (More senior @Release Managers: correct me if i am wrong)
Nevertheless, time to time, there seem to be a need for 'EOL'ing / deleting branches that are not in the releases repo's deliverables/* directory (for example: projects opened a stable branch manually; no tags / releases were administered; branches were reopened accidentally; given project was not part of the official series release; etc). These needs to be handled out of the releases repo 'deliverables' directory, thus needs different handling & tools. And this task is not really coupled with release management, and also note, that the release management team is a very small team nowadays.
Anyway, as a general stable maintainer core, I'm also thinking about how to solve this issue (old, obsolete branch deletion) in a safe way, though I'm not there yet to have any concrete solution. Ideas are welcome 🙂 . Until a solution is not found, the quickest solution is I think is to add the above ACL extension to the ironic.config [1].
Cheers,
Előd Illés irc: elodilles @ #openstack-release #openstack-stable
[1] https://opendev.org/openstack/project-config/src/commit/e92af53c10a811f4370c...
------------------------------ *From:* Jay Faulkner <jay@gr-oss.io> *Sent:* Tuesday, November 8, 2022 11:33 PM *To:* OpenStack Discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: [ironic][release] Bugfix branch status and cleanup w/r/t zuul-config-errors
Does anyone from the Releases team want to chime in on the best way to execute this kind of change?
-JayF
On Wed, Nov 2, 2022 at 7:02 AM Jay Faulkner <jay@gr-oss.io> wrote:
On Wed, Nov 2, 2022 at 3:04 AM Dmitry Tantsur <dtantsur@redhat.com> wrote:
Hi Jay,
On Tue, Nov 1, 2022 at 8:17 PM Jay Faulkner <jay@gr-oss.io> wrote:
Hey all,
I've been looking into the various zuul config errors showing up for Ironic-program branches. Almost all of our old bugfix branches are in the list. Additionally, not properly retiring the bugfix branches leads to an ever-growing list of branches which makes it a lot more difficult, for contributors and operators alike, to tell which ones are currently supported.
I'd like to see the errors. We update Zuul configuration manually for each bugfix branch, mapping appropriate branches for other projects (devstack, nova, etc). It's possible that we always overlook a few jobs, which causes Zuul to be upset (but quietly upset, so we don't notice).
The errors show up in https://zuul.opendev.org/t/openstack/config-errors -- although they seem to be broken this morning. Most of them are older bugfix branches, ones that are out of support, that have the `Queue: Ironic` param that's no longer supported. I am not in favor of anyone going to dead bugfix branches and fixing CI; instead we should retire the ones out of use.
I've put together a document describing the situation as it is now, and my proposal: https://etherpad.opendev.org/p/IronicBugfixBranchCleanup
Going with the "I would like to retire" would cause us so much trouble that we'll have to urgently create a downstream mirror of them. Once we do this, using upstream bugfix branches at all will be questionable. Especially bugfix/19.0 (and corresponding IPA/inspector branches) is used in a very actively maintained release.
Then we won't; but we do need to think about what timeline we can talk about upstream for getting a cadence for getting these retired out, just like we have a cadence for getting them cut every two months. I'll revise the list and remove the "I would like to retire" section (move it to keep-em-up).
Essentially, I think we need to: - identify bugfix branches to cleanup (I've done this in the above etherpad, but some of the ) - clean them up (the next step) - update Ironic policy to set a regular cadence for when to retire bugfix branches, and encode the process for doing so
This means there are two overall questions to answer in this email: 1) Mechanically, what's the process for doing this? I don't believe the existing release tooling will be useful for this, but I'm not 100% sure. I've pulled (in the above etherpad and a local spreadsheet) the last SHA for each branch; so we should be able to EOL these branches similarly to how we EOL stable branches; except manually instead of with tooling. Who is going to do this work? (I'd prefer releases team continue to hold the keys to do this; but I understand if you don't want to take on this manual work).
EOL tags will be created by the release team, yes. I don't think we can get the keys without going "independent".
It's a gerrit ACL you can enable to give other people access to tags; but like I said, I don't want that access anyway :).
2) What's the pattern for Ironic to adopt regarding these branches? We just need to write down the expected lifecycle and enforce it -- so we prevent being this deep into "branch debt" in the future.
With my vendor's (red) hat on, I'd prefer to have a dual approach: the newest branches are supported by the community (i.e. us all), the oldest - by vendors who need them (EOLed if nobody volunteers). I think you already have a list of branches that OCP uses? Feel free to point Riccardo, Iury or myself at any issues with them.
That's not really an option IMO. These branches exist in the upstream community, and are seen by upstream contributors and operators. If they're going to live here; they need to have some reasonable documentation about what folks should expect out of them and efforts being put towards them. Even if the documentation is "bugfix/1.2 is maintained as long as Product A 1.2 is maintained", that's better than leaving the community guessing about what these are used for, and why some are more-supported than others.
-Jay
Dmitry
What do folks think?
- Jay Faulkner
--
Red Hat GmbH <https://www.redhat.com/de/global/dach>, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill, Amy Ross
participants (4)
-
Dmitry Tantsur
-
Elõd Illés
-
Jay Faulkner
-
Jeremy Stanley