Open Stack

On Fri, Jan 26, 2018 at 5:57 PM James E. Blair <corvus at inaugust.com> wrote:

> Balázs Gibizer <balazs.gibizer at ericsson.com> writes:
>
> > Hi,
> >
> > I'm getting more and more confused how the zuul job hierarchy works or
> > is supposed to work.
>
> Hi!
>
> First, you (or others) may or may not have seen this already -- some of
> it didn't exist when we first rolled out v3, and some of it has changed
> -- but here are the relevant bits of the documentation that should help
> explain what's going on.  It helps to understand freezing:
>
>   https://docs.openstack.org/infra/zuul/user/config.html#job
>
> and matching:
>
>   https://docs.openstack.org/infra/zuul/user/config.html#matchers
>
> > First there was a bug in nova that some functional tests are not
> > triggered although the job (re-)definition in the nova part of the
> > project-config should not prevent it to run [1].
> >
> > There we figured out that irrelevant-files parameter of the jobs are
> > not something that can be overriden during re-definition or through
> > parent-child relationship. The base job openstack-tox-functional has
> > an irrelevant-files attribute that lists '^doc/.*$' as a path to be
> > ignored [2]. In the other hand the nova part of the project-config
> > tries to make this ignore less broad by adding only '^doc/source/.*$'
> > . This does not work as we expected and the job did not run on changes
> > that only affected ./doc/notification_samples path. We are fixing it
> > by defining our own functional job in nova tree [4].
> >
> > [1] https://bugs.launchpad.net/nova/+bug/1742962
> > [2]
> >
> https://github.com/openstack-infra/openstack-zuul-jobs/blob/1823e3ea20e6dfaf37786a6ff79c56cb786bf12c/zuul.d/jobs.yaml#L380
> > [3]
> >
> https://github.com/openstack-infra/project-config/blob/1145ab1293f5fa4d34c026856403c22b091e673c/zuul.d/projects.yaml#L10509
> > [4] https://review.openstack.org/#/c/533210/
>
> This is correct.  The issue here is that the irrelevant-files definition
> on openstack-tox-functional is too broad.  We need to be *extremely*
> careful applying matchers to jobs like that.  Generally I think that
> irrelevant-files should be reserved for the project-pipeline invocations
> only.  That's how they were effectively used in Zuul v2, after all.
>
> Essentially, when someone puts an irrelevant-files section on a job like
> that, they are saying "this job will never apply to these files, ever."
> That's clearly not correct in this case.
>
> So our solutions are to acknowledge that it's over-broad, and reduce or
> eliminate the list in [2] and expand it elsewhere (as in [3]).  Or we
> can say "we were generally correct, but nova is extra special so it
> needs its own job".  If that's the choice, then I think [4] is a fine
> solution.
>
> > Then I started looking into other jobs to see if we made similar
> > mistakes. I found two other examples in the nova related jobs where
> > redefining the irrelevant-files of a job caused problems. In these
> > examples nova tried to ignore more paths during the override than what
> > was originally ignored in the job definition but that did not work
> > [5][6].
> >
> > [5] https://bugs.launchpad.net/nova/+bug/1745405 (temptest-full)
>
> As noted in that bug, the tempest-full job is invoked on nova via this
> stanza:
>
>
> https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10674-L10688
>
> As expected, that did not match.  There is a second invocation of
> tempest-full on nova here:
>
>
> http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-project-templates.yaml#n126


I guess the line number changed since so this has moved to L101 [1] now :).
tempest-full is part of the integrated-gate, so all projects in it run it
through there.

[1]
http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-project-templates.yaml#n101


>
>
> That has no irrelevant-files matches, and so matches everything.  If you
> drop the use of that template, it will work as expected.  Or, if you can
> say with some certainty that nova's irrelevant-files set is not
> over-broad, you could move the irrelevant-files from nova's invocation
> into the template, or even the job, and drop nova's individual
> invocation.
>
> I don't think projects in the integrated gate should remove themselves
from the
template, it really helps keeping consistency.

The pattern I've seen is that most projects repeat the same list of
irrelevant files
over and over again in all of their integration tests, It would be handy in
future to
be able to set irrelevant-files on a template when it's consumed.
So we could have shared irrelevant files defined in the template, and
custom ones
added by each project when consuming the template. I don't this is is
possible today.
Does it sound like a reasonable feature request?

Andrea Frittoli (andreaf)


> > [6] https://bugs.launchpad.net/nova/+bug/1745431 (neutron-grenade)
>
> The same template invokes this job as well.
>
> > So far the problem seemed to be consistent (i.e. override does not
> > work). But then I looked into neutron-grenade-multinode. That job is
> > defined in neutron tree (like neutron-grenade) but nova also refers to
> > it in nova section of the project-config with different
> > irrelevant-files than their original definition. So I assumed that
> > this will lead to similar problem than in case of neutron-grenade, but
> > it doesn't.
> >
> > The neutron-grenade-multinode original definition [1] does not try to
> > ignore the 'nova/tests' path but the nova side of the definition in
> > the project config does try to ignore that path [8]. Interestingly a
> > patch in nova that only changes under the path: nova/tests/ does not
> > trigger the job [9]. So in this case overriding the irrelevant-files
> > of a job works. (It seems that overriding neutron-tempest-linuxbridge
> > irrelevant-files works too).
> >
> > [7]
> >
> https://github.com/openstack/neutron/blob/7e3d6a18fb928bcd303a44c1736d0d6ca9c7f0ab/.zuul.yaml#L140-L159
> > [8]
> >
> https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10516-L10530
> > [9] https://review.openstack.org/#/c/537936/
> >
> > I don't see what is the difference between neutron-grenade and
> > neutron-grenade-multinode jobs definitions from this perspective but
> > it seems that the irrelevent-files attribute behaves  inconsistently
> > in these two jobs. Could you please help me undestand how
> > irrelevant-files in overriden jobs supposed to work?
>
> These jobs only have the one invocation -- on the nova project -- and
> are not added via a template.
>
> Hopefully that explains the difference.
>
> Basically, the irrelevant-files on at least one project-pipeline
> invocation of a job have to match, as well as at least one definition of
> the job.  So if both things have irrelevant-files, then it's effectively
> a union of the two.
>
> I used a tool to help verify some of the information in this message,
> especially the bugs [5] and [6].  You can ask Zuul to output debug
> information about its job selection if you're dealing with confusing
> situations like this.  I went ahead and pushed a new patchset to your
> test change to demonstrate how:
>
>   https://review.openstack.org/537936
>
> When it finishes running all the tests (in a few hours), it should
> include in its report debug information about the decision-making
> process for the jobs it ran.  It outputs similar information into the
> debug logs; so that we don't have to wait for it to see what it looks
> like here is that copy:
>
>   http://paste.openstack.org/show/653729/
>
> The relevant lines for [5] are:
>
> 2018-01-26 13:07:53,560 DEBUG zuul.layout: Pipeline variant <Job
> tempest-full branches: None source:
> openstack-infra/openstack-zuul-jobs/zuul.d/zuul-legacy-project-templates.yaml at master#126>
> matched <Change 0x7f2fed8883c8 537936,2>
> 2018-01-26 13:07:53,560 DEBUG zuul.layout: Pipeline variant <Job
> tempest-full branches: None source:
> openstack-infra/project-config/zuul.d/projects.yaml at master#10485> did not
> match <Change 0x7f2fed8883c8 537936,2>
>
> Note the project-file-branch-line-number references are especially
> helpful.
>
> -Jim
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180214/68891909/attachment.html>