3rd party CI failures with devstack 'master' using devstack-gate

Ghanshyam Mann gmann at ghanshyammann.com
Tue Jun 15 14:37:42 UTC 2021


 ---- On Tue, 08 Jun 2021 12:12:11 -0500 Clark Boylan <cboylan at sapwetik.org> wrote ----
 > On Tue, Jun 8, 2021, at 7:14 AM, Ghanshyam Mann wrote:
 > >  ---- On Tue, 08 Jun 2021 07:42:21 -0500 Luigi Toscano 
 > > <ltoscano at redhat.com> wrote ----
 > >  > On Tuesday, 8 June 2021 14:11:40 CEST Fernando Ferraz wrote:
 > >  > > Hello,
 > >  > > 
 > >  > >  The NetApp CI for Cinder also relies on Zuul v2. We were able to
 > >  > > recently move our jobs to focal, but dropping devstack-gate is a 
 > > big
 > >  > > concern considering our team size and schedule.
 > >  > > Luigi, could you clarify what would immediately break after xena is
 > >  > > branched?
 > >  > > 
 > >  > 
 > >  > For example grenade jobs won't work anymore because there won't be 
 > > any new 
 > >  > entry related to stable/xena added here to devstack-vm-gate-wrap.sh:
 > >  > 
 > >  > 
 > > https://opendev.org/openstack/devstack-gate/src/branch/master/devstack-vm-gate-wrap.sh#L335
 > >  > 
 > >  > I understand that grenade testing is probably not relevant for 3rd 
 > > party CIs 
 > >  > (it should be, but that's a different discussion), but the main 
 > > point is that 
 > >  > devstack-gate is already now in almost-maintenance mode. The minimum 
 > > amount of 
 > >  > fixed that have been merged have been used to keep working the very 
 > > few legacy 
 > >  > jobs defined on opendev.org, and that number is basically 0 at this 
 > > point.
 > >  > 
 > >  > This mean that there are a ton of potential breakages happening 
 > > anytime, and 
 > >  > the focal change is just one (and each one of you, CI owner, had to 
 > > fix it on 
 > >  > your own). Others may come anytime and they won't be detected nor 
 > > investigated 
 > >  > anymore because we don't have de-facto legacy jobs around since 
 > > wallaby.
 > >  > 
 > >  > To summarize: if you use Zuul v2, you have been running for a long 
 > > while on an 
 > >  > unsupported software stack. The last tiny bits which could be used 
 > > on both 
 > >  > zuulv2 and zuulv3 in legacy mode to easy the transition are 
 > > unsupported too. 
 > >  > 
 > >  > This problem, I believe, has been communicated periodically by the 
 > > various 
 > >  > team and the time to migrate is... last month. Please hurry up!
 > > 
 > > Yes, we have done this migration in Victoria release cycle with two 
 > > community-wide goals together
 > > with the direction of moving all the CI from devstack gate from wallaby 
 > > itself. But by seeing few jobs
 > > and especially 3rd party CI, we extended the devstack-gate support for 
 > > wallaby release [1]. So we 
 > > extended the support for one more release until stable/wallaby.
 > > 
 > > NOTE: supporting a extra release extend the devstack-gate support until 
 > > that release until that become EOL,
 > > as we need to support that release stable CI. So it is not just a one 
 > > more cycle support but even longer
 > > time of 1 year or more.
 > > 
 > > Now extended the support for Xena cycle also seems very difficult by 
 > > seeing very less number of
 > > contributor or less bandwidth of current core members in devstack-gate.
 > > 
 > > I will plan to officially declare the devstack-gate deprecation with 
 > > team but please move your CI/CD to
 > > latest Focal and to zuulv3 ASAP.
 > 
 > These changes have started to go up [2].
 > 
 > I want to clarify a few things though. As far as I can remember we have never required any specific CI system or setup. What we have done are required basic behaviors from the CI system. Things like respond to "recheck", post logs in a publicly accessible location and report them back, have contacts available so we can contact you if things break, and so on. What this means is that some third party CI system are likely running Jenkins. I know others that ran some homegrown thing that watched the Gerrit event stream. We recommend Zuul and now Zuulv3 or newer because it is a tool that we understand and can provide some assistance with.
 > 
 > Those that choose not to use the recommended tools are likely to need to invest in their own tooling and debugging. For devstack-gate we will not accept new patches to keep it running against master, but need to keep it around for older stable branches. If those that are running their own set of tools want to keep devstack-gate alive for modern openstack then forking it is likely the best path forward.

Updates: All the patches for deprecating the devstack-gate are merged now, along with governance one:

- https://review.opendev.org/c/openstack/governance/+/795385

README file has been updated with the warning and about forking way Clark mentioned above:

https://opendev.org/openstack/devstack-gate/src/branch/master/README.rst

-gmann

 > 
 > > 
 > > 1. 
 > > https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jobs-to-ubuntu-focal.html
 > > 2. 
 > > https://governance.openstack.org/tc/goals/selected/victoria/native-zuulv3-jobs.html
 > > 
 > > 
 > > [1] 
 > > https://review.opendev.org/c/openstack/devstack-gate/+/778129
 > > https://review.opendev.org/c/openstack/devstack-gate/+/785010
 > 
 > [2] https://review.opendev.org/q/topic:%22deprecate-devstack-gate%22+(status:open%20OR%20status:merged)
 > 



More information about the openstack-discuss mailing list