On Wed, Nov 2, 2022 at 3:04 AM Dmitry Tantsur dtantsur@redhat.com wrote:

Hi Jay,

On Tue, Nov 1, 2022 at 8:17 PM Jay Faulkner jay@gr-oss.io wrote:

...

Hey all,

I've been looking into the various zuul config errors showing up for Ironic-program branches. Almost all of our old bugfix branches are in the list. Additionally, not properly retiring the bugfix branches leads to an ever-growing list of branches which makes it a lot more difficult, for contributors and operators alike, to tell which ones are currently supported.

I'd like to see the errors. We update Zuul configuration manually for each bugfix branch, mapping appropriate branches for other projects (devstack, nova, etc). It's possible that we always overlook a few jobs, which causes Zuul to be upset (but quietly upset, so we don't notice).

The errors show up in https://zuul.opendev.org/t/openstack/config-errors -- although they seem to be broken this morning. Most of them are older bugfix branches, ones that are out of support, that have the `Queue: Ironic` param that's no longer supported. I am not in favor of anyone going to dead bugfix branches and fixing CI; instead we should retire the ones out of use.

...

I've put together a document describing the situation as it is now, and my proposal: https://etherpad.opendev.org/p/IronicBugfixBranchCleanup

Going with the "I would like to retire" would cause us so much trouble that we'll have to urgently create a downstream mirror of them. Once we do this, using upstream bugfix branches at all will be questionable. Especially bugfix/19.0 (and corresponding IPA/inspector branches) is used in a very actively maintained release.

Then we won't; but we do need to think about what timeline we can talk about upstream for getting a cadence for getting these retired out, just like we have a cadence for getting them cut every two months. I'll revise the list and remove the "I would like to retire" section (move it to keep-em-up).

...

Essentially, I think we need to:

• identify bugfix branches to cleanup (I've done this in the above

etherpad, but some of the )

• clean them up (the next step)
• update Ironic policy to set a regular cadence for when to retire bugfix

branches, and encode the process for doing so

This means there are two overall questions to answer in this email:

1. Mechanically, what's the process for doing this? I don't believe the

existing release tooling will be useful for this, but I'm not 100% sure. I've pulled (in the above etherpad and a local spreadsheet) the last SHA for each branch; so we should be able to EOL these branches similarly to how we EOL stable branches; except manually instead of with tooling. Who is going to do this work? (I'd prefer releases team continue to hold the keys to do this; but I understand if you don't want to take on this manual work).

EOL tags will be created by the release team, yes. I don't think we can get the keys without going "independent".

It's a gerrit ACL you can enable to give other people access to tags; but like I said, I don't want that access anyway :).

...

2. What's the pattern for Ironic to adopt regarding these branches? We

just need to write down the expected lifecycle and enforce it -- so we prevent being this deep into "branch debt" in the future.

With my vendor's (red) hat on, I'd prefer to have a dual approach: the newest branches are supported by the community (i.e. us all), the oldest - by vendors who need them (EOLed if nobody volunteers). I think you already have a list of branches that OCP uses? Feel free to point Riccardo, Iury or myself at any issues with them.

That's not really an option IMO. These branches exist in the upstream community, and are seen by upstream contributors and operators. If they're going to live here; they need to have some reasonable documentation about what folks should expect out of them and efforts being put towards them. Even if the documentation is "bugfix/1.2 is maintained as long as Product A 1.2 is maintained", that's better than leaving the community guessing about what these are used for, and why some are more-supported than others.

-Jay

Dmitry

...

What do folks think?

Jay Faulkner

--

Red Hat GmbH https://www.redhat.com/de/global/dach, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill, Amy Ross

Hi,

TL;DR: for a quick solution I suggest to add the ACL [1] for the ironic cores, something like this:

[access "refs/heads/bugfix/*"] delete = = group ironic-core

The more lengthy answer: as far as I know, bugfix branches were allowed based on ironic team's request, but with the conditions, that

* it is out of Release Management scope * no official releases will be made out of these branches

So basically it's completely the Ironic team's responsibility. (More senior @Release Managers: correct me if i am wrong)

Nevertheless, time to time, there seem to be a need for 'EOL'ing / deleting branches that are not in the releases repo's deliverables/* directory (for example: projects opened a stable branch manually; no tags / releases were administered; branches were reopened accidentally; given project was not part of the official series release; etc). These needs to be handled out of the releases repo 'deliverables' directory, thus needs different handling & tools. And this task is not really coupled with release management, and also note, that the release management team is a very small team nowadays.

Anyway, as a general stable maintainer core, I'm also thinking about how to solve this issue (old, obsolete branch deletion) in a safe way, though I'm not there yet to have any concrete solution. Ideas are welcome 🙂 . Until a solution is not found, the quickest solution is I think is to add the above ACL extension to the ironic.config [1].

Cheers,

Előd Illés irc: elodilles @ #openstack-release #openstack-stable

[1] https://opendev.org/openstack/project-config/src/commit/e92af53c10a811f4370c...

________________________________ From: Jay Faulkner jay@gr-oss.io Sent: Tuesday, November 8, 2022 11:33 PM To: OpenStack Discuss openstack-discuss@lists.openstack.org Subject: Re: [ironic][release] Bugfix branch status and cleanup w/r/t zuul-config-errors

Does anyone from the Releases team want to chime in on the best way to execute this kind of change?

-JayF

On Wed, Nov 2, 2022 at 7:02 AM Jay Faulkner <jay@gr-oss.iomailto:jay@gr-oss.io> wrote:

On Wed, Nov 2, 2022 at 3:04 AM Dmitry Tantsur <dtantsur@redhat.commailto:dtantsur@redhat.com> wrote: Hi Jay,

On Tue, Nov 1, 2022 at 8:17 PM Jay Faulkner <jay@gr-oss.iomailto:jay@gr-oss.io> wrote: Hey all,

I've been looking into the various zuul config errors showing up for Ironic-program branches. Almost all of our old bugfix branches are in the list. Additionally, not properly retiring the bugfix branches leads to an ever-growing list of branches which makes it a lot more difficult, for contributors and operators alike, to tell which ones are currently supported.

I'd like to see the errors. We update Zuul configuration manually for each bugfix branch, mapping appropriate branches for other projects (devstack, nova, etc). It's possible that we always overlook a few jobs, which causes Zuul to be upset (but quietly upset, so we don't notice).

The errors show up in https://zuul.opendev.org/t/openstack/config-errors -- although they seem to be broken this morning. Most of them are older bugfix branches, ones that are out of support, that have the `Queue: Ironic` param that's no longer supported. I am not in favor of anyone going to dead bugfix branches and fixing CI; instead we should retire the ones out of use.

I've put together a document describing the situation as it is now, and my proposal: https://etherpad.opendev.org/p/IronicBugfixBranchCleanup

Going with the "I would like to retire" would cause us so much trouble that we'll have to urgently create a downstream mirror of them. Once we do this, using upstream bugfix branches at all will be questionable. Especially bugfix/19.0 (and corresponding IPA/inspector branches) is used in a very actively maintained release.

Then we won't; but we do need to think about what timeline we can talk about upstream for getting a cadence for getting these retired out, just like we have a cadence for getting them cut every two months. I'll revise the list and remove the "I would like to retire" section (move it to keep-em-up).

Essentially, I think we need to: - identify bugfix branches to cleanup (I've done this in the above etherpad, but some of the ) - clean them up (the next step) - update Ironic policy to set a regular cadence for when to retire bugfix branches, and encode the process for doing so

This means there are two overall questions to answer in this email: 1) Mechanically, what's the process for doing this? I don't believe the existing release tooling will be useful for this, but I'm not 100% sure. I've pulled (in the above etherpad and a local spreadsheet) the last SHA for each branch; so we should be able to EOL these branches similarly to how we EOL stable branches; except manually instead of with tooling. Who is going to do this work? (I'd prefer releases team continue to hold the keys to do this; but I understand if you don't want to take on this manual work).

EOL tags will be created by the release team, yes. I don't think we can get the keys without going "independent".

It's a gerrit ACL you can enable to give other people access to tags; but like I said, I don't want that access anyway :).

2) What's the pattern for Ironic to adopt regarding these branches? We just need to write down the expected lifecycle and enforce it -- so we prevent being this deep into "branch debt" in the future.

With my vendor's (red) hat on, I'd prefer to have a dual approach: the newest branches are supported by the community (i.e. us all), the oldest - by vendors who need them (EOLed if nobody volunteers). I think you already have a list of branches that OCP uses? Feel free to point Riccardo, Iury or myself at any issues with them.

That's not really an option IMO. These branches exist in the upstream community, and are seen by upstream contributors and operators. If they're going to live here; they need to have some reasonable documentation about what folks should expect out of them and efforts being put towards them. Even if the documentation is "bugfix/1.2 is maintained as long as Product A 1.2 is maintained", that's better than leaving the community guessing about what these are used for, and why some are more-supported than others.

-Jay

Dmitry

What do folks think?

- Jay Faulkner

--

Red Hat GmbHhttps://www.redhat.com/de/global/dach, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243, Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill, Amy Ross

Elod, thanks for the insight. We discussed this at Monday's Ironic meeting and decided to create a small, more limited group to gain this access.

I've taken the first step to enable this access via this merge request: https://review.opendev.org/admin/groups/0c53b8f80897aa9e7cee7347e4710bd9b8bd... -- once it's completed, I'll ask gerrit admins to give me access to modify membership. Once that's done, the Ironic cores who will be taking this access are me+our release liaisons, Dmitry, Iury, and Riccardo.

Thanks, Jay Faulkner

On Thu, Dec 1, 2022 at 10:04 AM Elõd Illés elod.illes@est.tech wrote:

Hi,

TL;DR: for a quick solution I suggest to add the ACL [1] for the ironic cores, something like this:

[access "refs/heads/bugfix/*"] delete = = group ironic-core

The more lengthy answer: as far as I know, bugfix branches were allowed based on ironic team's request, but with the conditions, that

• it is out of Release Management scope
• no official releases will be made out of these branches

So basically it's completely the Ironic team's responsibility. (More senior @Release Managers: correct me if i am wrong)

Nevertheless, time to time, there seem to be a need for 'EOL'ing / deleting branches that are not in the releases repo's deliverables/* directory (for example: projects opened a stable branch manually; no tags / releases were administered; branches were reopened accidentally; given project was not part of the official series release; etc). These needs to be handled out of the releases repo 'deliverables' directory, thus needs different handling & tools. And this task is not really coupled with release management, and also note, that the release management team is a very small team nowadays.

Anyway, as a general stable maintainer core, I'm also thinking about how to solve this issue (old, obsolete branch deletion) in a safe way, though I'm not there yet to have any concrete solution. Ideas are welcome 🙂 . Until a solution is not found, the quickest solution is I think is to add the above ACL extension to the ironic.config [1].

Cheers,

Előd Illés irc: elodilles @ #openstack-release #openstack-stable

[1] https://opendev.org/openstack/project-config/src/commit/e92af53c10a811f4370c...

---

*From:* Jay Faulkner jay@gr-oss.io *Sent:* Tuesday, November 8, 2022 11:33 PM *To:* OpenStack Discuss openstack-discuss@lists.openstack.org *Subject:* Re: [ironic][release] Bugfix branch status and cleanup w/r/t zuul-config-errors

Does anyone from the Releases team want to chime in on the best way to execute this kind of change?

-JayF

On Wed, Nov 2, 2022 at 7:02 AM Jay Faulkner jay@gr-oss.io wrote:

On Wed, Nov 2, 2022 at 3:04 AM Dmitry Tantsur dtantsur@redhat.com wrote:

Hi Jay,

On Tue, Nov 1, 2022 at 8:17 PM Jay Faulkner jay@gr-oss.io wrote:

Hey all,

I've been looking into the various zuul config errors showing up for Ironic-program branches. Almost all of our old bugfix branches are in the list. Additionally, not properly retiring the bugfix branches leads to an ever-growing list of branches which makes it a lot more difficult, for contributors and operators alike, to tell which ones are currently supported.

I'd like to see the errors. We update Zuul configuration manually for each bugfix branch, mapping appropriate branches for other projects (devstack, nova, etc). It's possible that we always overlook a few jobs, which causes Zuul to be upset (but quietly upset, so we don't notice).

The errors show up in https://zuul.opendev.org/t/openstack/config-errors -- although they seem to be broken this morning. Most of them are older bugfix branches, ones that are out of support, that have the `Queue: Ironic` param that's no longer supported. I am not in favor of anyone going to dead bugfix branches and fixing CI; instead we should retire the ones out of use.

I've put together a document describing the situation as it is now, and my proposal: https://etherpad.opendev.org/p/IronicBugfixBranchCleanup

Going with the "I would like to retire" would cause us so much trouble that we'll have to urgently create a downstream mirror of them. Once we do this, using upstream bugfix branches at all will be questionable. Especially bugfix/19.0 (and corresponding IPA/inspector branches) is used in a very actively maintained release.

Then we won't; but we do need to think about what timeline we can talk about upstream for getting a cadence for getting these retired out, just like we have a cadence for getting them cut every two months. I'll revise the list and remove the "I would like to retire" section (move it to keep-em-up).

Essentially, I think we need to:

• identify bugfix branches to cleanup (I've done this in the above

etherpad, but some of the )

• clean them up (the next step)
• update Ironic policy to set a regular cadence for when to retire bugfix

branches, and encode the process for doing so

This means there are two overall questions to answer in this email:

1. Mechanically, what's the process for doing this? I don't believe the

existing release tooling will be useful for this, but I'm not 100% sure. I've pulled (in the above etherpad and a local spreadsheet) the last SHA for each branch; so we should be able to EOL these branches similarly to how we EOL stable branches; except manually instead of with tooling. Who is going to do this work? (I'd prefer releases team continue to hold the keys to do this; but I understand if you don't want to take on this manual work).

EOL tags will be created by the release team, yes. I don't think we can get the keys without going "independent".

It's a gerrit ACL you can enable to give other people access to tags; but like I said, I don't want that access anyway :).

2. What's the pattern for Ironic to adopt regarding these branches? We

just need to write down the expected lifecycle and enforce it -- so we prevent being this deep into "branch debt" in the future.

With my vendor's (red) hat on, I'd prefer to have a dual approach: the newest branches are supported by the community (i.e. us all), the oldest - by vendors who need them (EOLed if nobody volunteers). I think you already have a list of branches that OCP uses? Feel free to point Riccardo, Iury or myself at any issues with them.

That's not really an option IMO. These branches exist in the upstream community, and are seen by upstream contributors and operators. If they're going to live here; they need to have some reasonable documentation about what folks should expect out of them and efforts being put towards them. Even if the documentation is "bugfix/1.2 is maintained as long as Product A 1.2 is maintained", that's better than leaving the community guessing about what these are used for, and why some are more-supported than others.

-Jay

Dmitry

What do folks think?

Jay Faulkner

--

Red Hat GmbH https://www.redhat.com/de/global/dach, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill, Amy Ross