[release] Independent release jobs are still using old release job templates
tl;dr: How do we get projects using the independent release process to use the openstack-python3 zuul job template corresponding to the latest release? I've noticed a couple of patches like [1] popping up for oslo-affiliated projects. These projects are currently using the python3 zuul job templates corresponding to an older release (e.g. Wallaby) and the author is correctly updating them to use the most recent release. These initially caught me off-guard since I'm used to seeing a bot propose these changes but I've realized that all affected projects are using the "independent" release process and therefore don't receive a new stable branch nor related bot-proposed commits. This leaves us in a bit of a pickle though. There are a lot of projects using this release process, and there's effectively zero chance that someone is going to remember to bump the jobs every single release.
While some could argue that this is "working as designed", it seems important to me that we would be testing these projects against the supported runtimes for the release currently under development. As such, how should we try to address this? One option I see is to add a new generic 'openstack-python3-jobs' template which would always duplicate the jobs used for the current release. I've proposed that here [2]. There are some pros and cons to this approach:
* PRO: Minimal effort from everyone. * CON: The HEAD of master for each project could be broken and we'd never know until some change was proposed to master (note: this is effectively the status quo right now, so it's not a big break) * CON: If those independent projects had an alternative to stable branches (i.e. semver branches), those would eventually break as the runtimes kept changing.
The other option is to extend the openstack bot to propose release bump patches against these independent projects. This has the inverse pro/con balance:
* PRO: We get a regular "check-up" on the health of the project whenever these patches are proposed. * PRO: Long-lived stable branches should stay working * CON: Lots of busy work for reviewers.
What are everyone's thoughts? I'm in favour of the first approach (generic 'openstack-python3-jobs' template) but the bot enhancement would also wfm. If we go with the first approach, we'd probably want to script something to "fix" all the existing independent projects rather than attempt to find and fix them manually.
Cheers, Stephen
PS: Apologies if this has been discussed previously. I clearly missed that discussion.
[1] https://review.opendev.org/c/openstack/os-api-ref/+/831560 [2] https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/833286
On 2022-03-11 12:26:38 +0000 (+0000), Stephen Finucane wrote: [...]
These initially caught me off-guard since I'm used to seeing a bot propose these changes but I've realized that all affected projects are using the "independent" release process and therefore don't receive a new stable branch nor related bot-proposed commits.
[...]
It's more than this, the "independent" release model definition is that projects have opted out of central release management entirely:
https://releases.openstack.org/reference/release_models.html#independent
If you want the OpenStack release managers coordinating cycle activities for those projects, I suspect we may need to redefine the relationship there.
On Fri, 2022-03-11 at 13:38 +0000, Jeremy Stanley wrote:
On 2022-03-11 12:26:38 +0000 (+0000), Stephen Finucane wrote: [...]
These initially caught me off-guard since I'm used to seeing a bot propose these changes but I've realized that all affected projects are using the "independent" release process and therefore don't receive a new stable branch nor related bot-proposed commits.
[...]
It's more than this, the "independent" release model definition is that projects have opted out of central release management entirely:
https://releases.openstack.org/reference/release_models.html#independent
If you want the OpenStack release managers coordinating cycle activities for those projects, I suspect we may need to redefine the relationship there.
Yes, this would be a big disadvantage of the second option I proposed. I suspect "option A" will allow us to maintain the existing relationship with the sole change of "we can now potentially break your project's CI when we bump the runtimes used by the unversioned job template at the start of a cycle (but you can opt out by using the versioned job templates)". This seems a reasonably uncontroversial change from my perspective, particularly given the alternatives.
Stephen
Hello,
For now +1 for the option A.
The goal of moving those Oslo deliverables as independent deliverables was to reduce our workload so instead of voting for a time consuming option (the option B) I'd rather suggest an option C that would be moving back those deliverables as coordinated deliverables.
This is not the first time that switching those deliverables to the independent model causes us problems.
Thoughts?
Le lun. 14 mars 2022 à 13:08, Stephen Finucane stephenfin@redhat.com a écrit :
On Fri, 2022-03-11 at 13:38 +0000, Jeremy Stanley wrote:
On 2022-03-11 12:26:38 +0000 (+0000), Stephen Finucane wrote: [...]
These initially caught me off-guard since I'm used to seeing a bot propose these changes but I've realized that all affected projects are using the "independent" release process and therefore don't receive a new stable branch nor related bot-proposed commits.
[...]
It's more than this, the "independent" release model definition is that projects have opted out of central release management entirely:
https://releases.openstack.org/reference/release_models.html#independent
If you want the OpenStack release managers coordinating cycle activities for those projects, I suspect we may need to redefine the relationship there.
Yes, this would be a big disadvantage of the second option I proposed. I suspect "option A" will allow us to maintain the existing relationship with the sole change of "we can now potentially break your project's CI when we bump the runtimes used by the unversioned job template at the start of a cycle (but you can opt out by using the versioned job templates)". This seems a reasonably uncontroversial change from my perspective, particularly given the alternatives.
Stephen
On Fri, 2022-03-11 at 12:26 +0000, Stephen Finucane wrote:
tl;dr: How do we get projects using the independent release process to use the openstack-python3 zuul job template corresponding to the latest release?
FYI, we've opted to go with the first option below (introducing an unversioned job template that projects using the independent release process can use). This was achieved via [1] which should merge shortly. I've pushed a number of patches to fix this for various oslo- and sdk-affiliated projects [2]. I trust other teams (puppet? charms?) can handle this for their own projects :)
Cheers, Stephen
[1] https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/833286 [2] https://review.opendev.org/q/topic:add-python3-unversioned-jobs
I've noticed a couple of patches like [1] popping up for oslo-affiliated projects. These projects are currently using the python3 zuul job templates corresponding to an older release (e.g. Wallaby) and the author is correctly updating them to use the most recent release. These initially caught me off-guard since I'm used to seeing a bot propose these changes but I've realized that all affected projects are using the "independent" release process and therefore don't receive a new stable branch nor related bot-proposed commits. This leaves us in a bit of a pickle though. There are a lot of projects using this release process, and there's effectively zero chance that someone is going to remember to bump the jobs every single release.
While some could argue that this is "working as designed", it seems important to me that we would be testing these projects against the supported runtimes for the release currently under development. As such, how should we try to address this? One option I see is to add a new generic 'openstack-python3-jobs' template which would always duplicate the jobs used for the current release. I've proposed that here [2]. There are some pros and cons to this approach:
- PRO: Minimal effort from everyone.
- CON: The HEAD of master for each project could be broken and we'd never know until some change was proposed to master (note: this is effectively the status quo right now, so it's not a big break)
- CON: If those independent projects had an alternative to stable branches (i.e. semver branches), those would eventually break as the runtimes kept changing.
The other option is to extend the openstack bot to propose release bump patches against these independent projects. This has the inverse pro/con balance:
- PRO: We get a regular "check-up" on the health of the project whenever these patches are proposed.
- PRO: Long-lived stable branches should stay working
- CON: Lots of busy work for reviewers.
What are everyone's thoughts? I'm in favour of the first approach (generic 'openstack-python3-jobs' template) but the bot enhancement would also wfm. If we go with the first approach, we'd probably want to script something to "fix" all the existing independent projects rather than attempt to find and fix them manually.
Cheers, Stephen
PS: Apologies if this has been discussed previously. I clearly missed that discussion.
[1] https://review.opendev.org/c/openstack/os-api-ref/+/831560 [2] https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/833286
---- On Tue, 15 Mar 2022 11:29:45 -0500 Stephen Finucane stephenfin@redhat.com wrote ----
On Fri, 2022-03-11 at 12:26 +0000, Stephen Finucane wrote:
tl;dr: How do we get projects using the independent release process to use the openstack-python3 zuul job template corresponding to the latest release?
FYI, we've opted to go with the first option below (introducing an unversioned job template that projects using the independent release process can use). This was achieved via [1] which should merge shortly. I've pushed a number of patches to fix this for various oslo- and sdk-affiliated projects [2]. I trust other teams (puppet? charms?) can handle this for their own projects :)
Sorry for seeing this thread late.
I think testing with latest is no different than updating them manually to the latest release template either by bot or manually. As we use the constraint and add a compatible version of these independent repos for each OpenStack release, I don't think we have any case where new future version is not tested with a supported runtime where that version will be used/documented as the compatible version for XYZ OpenStack release. So +1 on option1.
Now thinking on this case also, these release-specific templates are causing more issues. For example: 1. This case of independent releases repo 2. If bot missed proposing patches for some reason (like format. other release model or so) or patches are not merged then we missed testing such repo with defined testing runtime. 3. branchless repo like PBR, Tempest has to test py version explicitly as they cannot use these release-wise templates.
With all these issues should we just get rid of these release-specific templates and have this generic template 'openstack-python3-jobs' with the latest testing runtime for master code testing and when we cut the stable branch then we pin a version of this template with help of the branch variant in the template definition (no list of jobs in that template as we cannot use branch variant for template). It means we are cutting a new stable for OpenStack so let's pin the testing template also for that branch (pin the same template as openstack-zuul-jobs are branchless) and branch variant can help us here. And for the new master, move the current template to the latest testing runtime. This helps to avoid all these bot patches and missing the latest testing running in the above-mentioned cases. One drawback for this is we have to remember and make sure we pin this same template when cut stable. But that effort is no different than defining the new template per release each time.
We made these release-specific templates to be very explicit in testing but it leads to testing old testing runtime due to mentioned issues. Making them controllable in a central place helps to avoid that and also avoid these bot patches and reviews in each repo.
WIP Example: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/833892
Should we re-consider that option?
-gmann
Cheers, Stephen
[1] https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/833286 [2] https://review.opendev.org/q/topic:add-python3-unversioned-jobs
I've noticed a couple of patches like [1] popping up for oslo-affiliated projects. These projects are currently using the python3 zuul job templates corresponding to an older release (e.g. Wallaby) and the author is correctly updating them to use the most recent release. These initially caught me off-guard since I'm used to seeing a bot propose these changes but I've realized that all affected projects are using the "independent" release process and therefore don't receive a new stable branch nor related bot-proposed commits. This leaves us in a bit of a pickle though. There are a lot of projects using this release process, and there's effectively zero chance that someone is going to remember to bump the jobs every single release.
While some could argue that this is "working as designed", it seems important to me that we would be testing these projects against the supported runtimes for the release currently under development. As such, how should we try to address this? One option I see is to add a new generic 'openstack-python3-jobs' template which would always duplicate the jobs used for the current release. I've proposed that here [2]. There are some pros and cons to this approach:
- PRO: Minimal effort from everyone.
- CON: The HEAD of master for each project could be broken and we'd never know until some change was proposed to master (note: this is effectively the status quo right now, so it's not a big break)
- CON: If those independent projects had an alternative to stable branches (i.e. semver branches), those would eventually break as the runtimes kept changing.
The other option is to extend the openstack bot to propose release bump patches against these independent projects. This has the inverse pro/con balance:
- PRO: We get a regular "check-up" on the health of the project whenever these patches are proposed.
- PRO: Long-lived stable branches should stay working
- CON: Lots of busy work for reviewers.
What are everyone's thoughts? I'm in favour of the first approach (generic 'openstack-python3-jobs' template) but the bot enhancement would also wfm. If we go with the first approach, we'd probably want to script something to "fix" all the existing independent projects rather than attempt to find and fix them manually.
Cheers, Stephen
PS: Apologies if this has been discussed previously. I clearly missed that discussion.
[1] https://review.opendev.org/c/openstack/os-api-ref/+/831560 [2] https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/833286
On 2022-03-15 12:50:06 -0500 (-0500), Ghanshyam Mann wrote: [...]
With all these issues should we just get rid of these release-specific templates and have this generic template 'openstack-python3-jobs' with the latest testing runtime for master code testing and when we cut the stable branch then we pin a version of this template with help of the branch variant in the template definition (no list of jobs in that template as we cannot use branch variant for template).
[...]
The flip side of that coin is that every project has to switch to the new jobs in master at the exact same moment, whether they're ready or not, so they'll need to be prepared to set aside time to sort out job failures before proceeding with their usual development.
---- On Tue, 15 Mar 2022 15:04:21 -0500 Jeremy Stanley fungi@yuggoth.org wrote ----
On 2022-03-15 12:50:06 -0500 (-0500), Ghanshyam Mann wrote: [...]
With all these issues should we just get rid of these release-specific templates and have this generic template 'openstack-python3-jobs' with the latest testing runtime for master code testing and when we cut the stable branch then we pin a version of this template with help of the branch variant in the template definition (no list of jobs in that template as we cannot use branch variant for template).
[...]
The flip side of that coin is that every project has to switch to the new jobs in master at the exact same moment, whether they're ready or not, so they'll need to be prepared to set aside time to sort out job failures before proceeding with their usual development.
That is a good thing right, fix the things for new python version testing before you introduce more failure. And in the current template change by bot is also at the start of the cycle and it usually gets merged before any other code on that master branch merge which means no change in timing as such.
-gmann
-- Jeremy Stanley
participants (4)
-
Ghanshyam Mann
-
Herve Beraud
-
Jeremy Stanley
-
Stephen Finucane