Re: 3rd party CI failures with devstack 'master' using devstack-gate
On Tuesday, 8 June 2021 11:50:32 CEST Katari Kumar wrote:
Hi,
our 3rd party CI (IBM Storage CI, based on zuul v2) uses devstack-gate scripts to install openstack via devstack and run tempest suite on the storage. It works with wallaby but fails on latest master as the devstack project dropped bionic support. We are currently trying to use ubuntu focal, but facing issues in devstack gate script. I understand that all 3rdparty drivers should migrate to Zuul v3 to avoid such issues. As devstack-gate is not used in Zuul V3 , i see no activity in devstack-gate to support latest versions. But as there are many existing Zuul v2 users, devstack-gate should continue to support latest projects.
This has been communicated several times: devstack-gate should have been dropped in ussuri already according the original plan. The plan was delayed a bit because we had a few relevant legacy jobs around, but the last bits have been merged recently and there are no further plans to support devstack-gate for xena. On your specific issue: I think we had a few focal-based legacy jobs in victoria before dropping them, so you may probably tune the jobs to work with devstack-gate. But this won't work when Xena is branched. So please prioritize the migration to Zuul v3, rather than trying to patch an unsupported software stack. During last PTG, in the Cinder session a 3rd party CI shared their experience with the migration using Software Factory as Zuul distribution, you can find the recording here: https://www.youtube.com/watch?v=hVLpPBldn7g&t=426 https://wiki.openstack.org/wiki/ CinderXenaPTGSummary#Using_Software_Factory_for_Cinder_Third_Party_CI -- Luigi
Hello, The NetApp CI for Cinder also relies on Zuul v2. We were able to recently move our jobs to focal, but dropping devstack-gate is a big concern considering our team size and schedule. Luigi, could you clarify what would immediately break after xena is branched? Fernando Em ter., 8 de jun. de 2021 às 07:05, Luigi Toscano <ltoscano@redhat.com> escreveu:
On Tuesday, 8 June 2021 11:50:32 CEST Katari Kumar wrote:
Hi,
our 3rd party CI (IBM Storage CI, based on zuul v2) uses devstack-gate scripts to install openstack via devstack and run tempest suite on the storage. It works with wallaby but fails on latest master as the devstack project dropped bionic support. We are currently trying to use ubuntu focal, but facing issues in
devstack
gate script. I understand that all 3rdparty drivers should migrate to Zuul v3 to avoid such issues. As devstack-gate is not used in Zuul V3 , i see no activity in devstack-gate to support latest versions. But as there are many existing Zuul v2 users, devstack-gate should continue to support latest projects.
This has been communicated several times: devstack-gate should have been dropped in ussuri already according the original plan. The plan was delayed a bit because we had a few relevant legacy jobs around, but the last bits have been merged recently and there are no further plans to support devstack-gate for xena.
On your specific issue: I think we had a few focal-based legacy jobs in victoria before dropping them, so you may probably tune the jobs to work with devstack-gate. But this won't work when Xena is branched.
So please prioritize the migration to Zuul v3, rather than trying to patch an unsupported software stack. During last PTG, in the Cinder session a 3rd party CI shared their experience with the migration using Software Factory as Zuul distribution, you can find the recording here:
https://www.youtube.com/watch?v=hVLpPBldn7g&t=426
https://wiki.openstack.org/wiki/ CinderXenaPTGSummary#Using_Software_Factory_for_Cinder_Third_Party_CI
-- Luigi
On Tuesday, 8 June 2021 14:11:40 CEST Fernando Ferraz wrote:
Hello,
The NetApp CI for Cinder also relies on Zuul v2. We were able to recently move our jobs to focal, but dropping devstack-gate is a big concern considering our team size and schedule. Luigi, could you clarify what would immediately break after xena is branched?
For example grenade jobs won't work anymore because there won't be any new entry related to stable/xena added here to devstack-vm-gate-wrap.sh: https://opendev.org/openstack/devstack-gate/src/branch/master/devstack-vm-ga... I understand that grenade testing is probably not relevant for 3rd party CIs (it should be, but that's a different discussion), but the main point is that devstack-gate is already now in almost-maintenance mode. The minimum amount of fixed that have been merged have been used to keep working the very few legacy jobs defined on opendev.org, and that number is basically 0 at this point. This mean that there are a ton of potential breakages happening anytime, and the focal change is just one (and each one of you, CI owner, had to fix it on your own). Others may come anytime and they won't be detected nor investigated anymore because we don't have de-facto legacy jobs around since wallaby. To summarize: if you use Zuul v2, you have been running for a long while on an unsupported software stack. The last tiny bits which could be used on both zuulv2 and zuulv3 in legacy mode to easy the transition are unsupported too. This problem, I believe, has been communicated periodically by the various team and the time to migrate is... last month. Please hurry up! Ciao -- Luigi
---- On Tue, 08 Jun 2021 07:42:21 -0500 Luigi Toscano <ltoscano@redhat.com> wrote ----
On Tuesday, 8 June 2021 14:11:40 CEST Fernando Ferraz wrote:
Hello,
The NetApp CI for Cinder also relies on Zuul v2. We were able to recently move our jobs to focal, but dropping devstack-gate is a big concern considering our team size and schedule. Luigi, could you clarify what would immediately break after xena is branched?
For example grenade jobs won't work anymore because there won't be any new entry related to stable/xena added here to devstack-vm-gate-wrap.sh:
https://opendev.org/openstack/devstack-gate/src/branch/master/devstack-vm-ga...
I understand that grenade testing is probably not relevant for 3rd party CIs (it should be, but that's a different discussion), but the main point is that devstack-gate is already now in almost-maintenance mode. The minimum amount of fixed that have been merged have been used to keep working the very few legacy jobs defined on opendev.org, and that number is basically 0 at this point.
This mean that there are a ton of potential breakages happening anytime, and the focal change is just one (and each one of you, CI owner, had to fix it on your own). Others may come anytime and they won't be detected nor investigated anymore because we don't have de-facto legacy jobs around since wallaby.
To summarize: if you use Zuul v2, you have been running for a long while on an unsupported software stack. The last tiny bits which could be used on both zuulv2 and zuulv3 in legacy mode to easy the transition are unsupported too.
This problem, I believe, has been communicated periodically by the various team and the time to migrate is... last month. Please hurry up!
Yes, we have done this migration in Victoria release cycle with two community-wide goals together with the direction of moving all the CI from devstack gate from wallaby itself. But by seeing few jobs and especially 3rd party CI, we extended the devstack-gate support for wallaby release [1]. So we extended the support for one more release until stable/wallaby. NOTE: supporting a extra release extend the devstack-gate support until that release until that become EOL, as we need to support that release stable CI. So it is not just a one more cycle support but even longer time of 1 year or more. Now extended the support for Xena cycle also seems very difficult by seeing very less number of contributor or less bandwidth of current core members in devstack-gate. I will plan to officially declare the devstack-gate deprecation with team but please move your CI/CD to latest Focal and to zuulv3 ASAP. 1. https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... 2. https://governance.openstack.org/tc/goals/selected/victoria/native-zuulv3-jo... [1] https://review.opendev.org/c/openstack/devstack-gate/+/778129 https://review.opendev.org/c/openstack/devstack-gate/+/785010 -gmann
Ciao -- Luigi
On Tue, Jun 8, 2021, at 7:14 AM, Ghanshyam Mann wrote:
---- On Tue, 08 Jun 2021 07:42:21 -0500 Luigi Toscano <ltoscano@redhat.com> wrote ----
On Tuesday, 8 June 2021 14:11:40 CEST Fernando Ferraz wrote:
Hello,
The NetApp CI for Cinder also relies on Zuul v2. We were able to recently move our jobs to focal, but dropping devstack-gate is a big concern considering our team size and schedule. Luigi, could you clarify what would immediately break after xena is branched?
For example grenade jobs won't work anymore because there won't be any new entry related to stable/xena added here to devstack-vm-gate-wrap.sh:
https://opendev.org/openstack/devstack-gate/src/branch/master/devstack-vm-ga...
I understand that grenade testing is probably not relevant for 3rd
(it should be, but that's a different discussion), but the main
devstack-gate is already now in almost-maintenance mode. The minimum amount of fixed that have been merged have been used to keep working the very few legacy jobs defined on opendev.org, and that number is basically 0 at this
party CIs point is that point.
This mean that there are a ton of potential breakages happening
anytime, and
the focal change is just one (and each one of you, CI owner, had to fix it on your own). Others may come anytime and they won't be detected nor investigated anymore because we don't have de-facto legacy jobs around since wallaby.
To summarize: if you use Zuul v2, you have been running for a long while on an unsupported software stack. The last tiny bits which could be used on both zuulv2 and zuulv3 in legacy mode to easy the transition are unsupported too.
This problem, I believe, has been communicated periodically by the various team and the time to migrate is... last month. Please hurry up!
Yes, we have done this migration in Victoria release cycle with two community-wide goals together with the direction of moving all the CI from devstack gate from wallaby itself. But by seeing few jobs and especially 3rd party CI, we extended the devstack-gate support for wallaby release [1]. So we extended the support for one more release until stable/wallaby.
NOTE: supporting a extra release extend the devstack-gate support until that release until that become EOL, as we need to support that release stable CI. So it is not just a one more cycle support but even longer time of 1 year or more.
Now extended the support for Xena cycle also seems very difficult by seeing very less number of contributor or less bandwidth of current core members in devstack-gate.
I will plan to officially declare the devstack-gate deprecation with team but please move your CI/CD to latest Focal and to zuulv3 ASAP.
These changes have started to go up [2]. I want to clarify a few things though. As far as I can remember we have never required any specific CI system or setup. What we have done are required basic behaviors from the CI system. Things like respond to "recheck", post logs in a publicly accessible location and report them back, have contacts available so we can contact you if things break, and so on. What this means is that some third party CI system are likely running Jenkins. I know others that ran some homegrown thing that watched the Gerrit event stream. We recommend Zuul and now Zuulv3 or newer because it is a tool that we understand and can provide some assistance with. Those that choose not to use the recommended tools are likely to need to invest in their own tooling and debugging. For devstack-gate we will not accept new patches to keep it running against master, but need to keep it around for older stable branches. If those that are running their own set of tools want to keep devstack-gate alive for modern openstack then forking it is likely the best path forward.
1. https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... 2. https://governance.openstack.org/tc/goals/selected/victoria/native-zuulv3-jo...
[1] https://review.opendev.org/c/openstack/devstack-gate/+/778129 https://review.opendev.org/c/openstack/devstack-gate/+/785010
[2] https://review.opendev.org/q/topic:%22deprecate-devstack-gate%22+(status:ope...)
---- On Tue, 08 Jun 2021 12:12:11 -0500 Clark Boylan <cboylan@sapwetik.org> wrote ----
On Tue, Jun 8, 2021, at 7:14 AM, Ghanshyam Mann wrote:
---- On Tue, 08 Jun 2021 07:42:21 -0500 Luigi Toscano <ltoscano@redhat.com> wrote ----
On Tuesday, 8 June 2021 14:11:40 CEST Fernando Ferraz wrote:
Hello,
The NetApp CI for Cinder also relies on Zuul v2. We were able to recently move our jobs to focal, but dropping devstack-gate is a big concern considering our team size and schedule. Luigi, could you clarify what would immediately break after xena is branched?
For example grenade jobs won't work anymore because there won't be any new entry related to stable/xena added here to devstack-vm-gate-wrap.sh:
https://opendev.org/openstack/devstack-gate/src/branch/master/devstack-vm-ga...
I understand that grenade testing is probably not relevant for 3rd
(it should be, but that's a different discussion), but the main
devstack-gate is already now in almost-maintenance mode. The minimum amount of fixed that have been merged have been used to keep working the very few legacy jobs defined on opendev.org, and that number is basically 0 at this
party CIs point is that point.
This mean that there are a ton of potential breakages happening
anytime, and
the focal change is just one (and each one of you, CI owner, had to fix it on your own). Others may come anytime and they won't be detected nor investigated anymore because we don't have de-facto legacy jobs around since wallaby.
To summarize: if you use Zuul v2, you have been running for a long while on an unsupported software stack. The last tiny bits which could be used on both zuulv2 and zuulv3 in legacy mode to easy the transition are unsupported too.
This problem, I believe, has been communicated periodically by the various team and the time to migrate is... last month. Please hurry up!
Yes, we have done this migration in Victoria release cycle with two community-wide goals together with the direction of moving all the CI from devstack gate from wallaby itself. But by seeing few jobs and especially 3rd party CI, we extended the devstack-gate support for wallaby release [1]. So we extended the support for one more release until stable/wallaby.
NOTE: supporting a extra release extend the devstack-gate support until that release until that become EOL, as we need to support that release stable CI. So it is not just a one more cycle support but even longer time of 1 year or more.
Now extended the support for Xena cycle also seems very difficult by seeing very less number of contributor or less bandwidth of current core members in devstack-gate.
I will plan to officially declare the devstack-gate deprecation with team but please move your CI/CD to latest Focal and to zuulv3 ASAP.
These changes have started to go up [2].
I want to clarify a few things though. As far as I can remember we have never required any specific CI system or setup. What we have done are required basic behaviors from the CI system. Things like respond to "recheck", post logs in a publicly accessible location and report them back, have contacts available so we can contact you if things break, and so on. What this means is that some third party CI system are likely running Jenkins. I know others that ran some homegrown thing that watched the Gerrit event stream. We recommend Zuul and now Zuulv3 or newer because it is a tool that we understand and can provide some assistance with.
Those that choose not to use the recommended tools are likely to need to invest in their own tooling and debugging. For devstack-gate we will not accept new patches to keep it running against master, but need to keep it around for older stable branches. If those that are running their own set of tools want to keep devstack-gate alive for modern openstack then forking it is likely the best path forward.
Updates: All the patches for deprecating the devstack-gate are merged now, along with governance one: - https://review.opendev.org/c/openstack/governance/+/795385 README file has been updated with the warning and about forking way Clark mentioned above: https://opendev.org/openstack/devstack-gate/src/branch/master/README.rst -gmann
1. https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... 2. https://governance.openstack.org/tc/goals/selected/victoria/native-zuulv3-jo...
[1] https://review.opendev.org/c/openstack/devstack-gate/+/778129 https://review.opendev.org/c/openstack/devstack-gate/+/785010
[2] https://review.opendev.org/q/topic:%22deprecate-devstack-gate%22+(status:ope...)
On 2021-06-08 14:42:21 +0200 (+0200), Luigi Toscano wrote: [...]
To summarize: if you use Zuul v2, you have been running for a long while on an unsupported software stack. The last tiny bits which could be used on both zuulv2 and zuulv3 in legacy mode to easy the transition are unsupported too. [...]
For very large definitions of "long while." The last official 2.x release of Zuul was in September of 2017, so it's been EOL going on 4 years already. I'm not sure how much more warning people need that they should upgrade? -- Jeremy Stanley
On 2021-06-08 09:50:32 +0000 (+0000), Katari Kumar wrote: [...]
But as there are many existing Zuul v2 users, devstack-gate should continue to support latest projects.
Community software is developed and supported by its users, and devstack-gate is no exception. The people who were maintaining it no longer have any use for it. If you're using it, then it's up to you to keep it working (perhaps with the help of others who are also using it). But in my biased opinion, your time is probably better spent upgrading than trying to limp along with abandonware. -- Jeremy Stanley
participants (6)
-
Clark Boylan
-
Fernando Ferraz
-
Ghanshyam Mann
-
Jeremy Stanley
-
Katari Kumar
-
Luigi Toscano