Open Stack

Balázs Gibizer <balazs.gibizer at ericsson.com> writes:

> Hi,
>
> I'm getting more and more confused how the zuul job hierarchy works or
> is supposed to work.

Hi!

First, you (or others) may or may not have seen this already -- some of
it didn't exist when we first rolled out v3, and some of it has changed
-- but here are the relevant bits of the documentation that should help
explain what's going on.  It helps to understand freezing:

  https://docs.openstack.org/infra/zuul/user/config.html#job

and matching:

  https://docs.openstack.org/infra/zuul/user/config.html#matchers

> First there was a bug in nova that some functional tests are not
> triggered although the job (re-)definition in the nova part of the
> project-config should not prevent it to run [1].
>
> There we figured out that irrelevant-files parameter of the jobs are
> not something that can be overriden during re-definition or through
> parent-child relationship. The base job openstack-tox-functional has
> an irrelevant-files attribute that lists '^doc/.*$' as a path to be
> ignored [2]. In the other hand the nova part of the project-config
> tries to make this ignore less broad by adding only '^doc/source/.*$'
> . This does not work as we expected and the job did not run on changes
> that only affected ./doc/notification_samples path. We are fixing it
> by defining our own functional job in nova tree [4].
>
> [1] https://bugs.launchpad.net/nova/+bug/1742962
> [2]
> https://github.com/openstack-infra/openstack-zuul-jobs/blob/1823e3ea20e6dfaf37786a6ff79c56cb786bf12c/zuul.d/jobs.yaml#L380
> [3]
> https://github.com/openstack-infra/project-config/blob/1145ab1293f5fa4d34c026856403c22b091e673c/zuul.d/projects.yaml#L10509
> [4] https://review.openstack.org/#/c/533210/

This is correct.  The issue here is that the irrelevant-files definition
on openstack-tox-functional is too broad.  We need to be *extremely*
careful applying matchers to jobs like that.  Generally I think that
irrelevant-files should be reserved for the project-pipeline invocations
only.  That's how they were effectively used in Zuul v2, after all.

Essentially, when someone puts an irrelevant-files section on a job like
that, they are saying "this job will never apply to these files, ever."
That's clearly not correct in this case.

So our solutions are to acknowledge that it's over-broad, and reduce or
eliminate the list in [2] and expand it elsewhere (as in [3]).  Or we
can say "we were generally correct, but nova is extra special so it
needs its own job".  If that's the choice, then I think [4] is a fine
solution.

> Then I started looking into other jobs to see if we made similar
> mistakes. I found two other examples in the nova related jobs where
> redefining the irrelevant-files of a job caused problems. In these
> examples nova tried to ignore more paths during the override than what
> was originally ignored in the job definition but that did not work
> [5][6].
>
> [5] https://bugs.launchpad.net/nova/+bug/1745405 (temptest-full)

As noted in that bug, the tempest-full job is invoked on nova via this
stanza:

https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10674-L10688

As expected, that did not match.  There is a second invocation of
tempest-full on nova here:

http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-project-templates.yaml#n126

That has no irrelevant-files matches, and so matches everything.  If you
drop the use of that template, it will work as expected.  Or, if you can
say with some certainty that nova's irrelevant-files set is not
over-broad, you could move the irrelevant-files from nova's invocation
into the template, or even the job, and drop nova's individual
invocation.

> [6] https://bugs.launchpad.net/nova/+bug/1745431 (neutron-grenade)

The same template invokes this job as well.

> So far the problem seemed to be consistent (i.e. override does not
> work). But then I looked into neutron-grenade-multinode. That job is
> defined in neutron tree (like neutron-grenade) but nova also refers to
> it in nova section of the project-config with different
> irrelevant-files than their original definition. So I assumed that
> this will lead to similar problem than in case of neutron-grenade, but
> it doesn't.
>
> The neutron-grenade-multinode original definition [1] does not try to
> ignore the 'nova/tests' path but the nova side of the definition in
> the project config does try to ignore that path [8]. Interestingly a
> patch in nova that only changes under the path: nova/tests/ does not
> trigger the job [9]. So in this case overriding the irrelevant-files
> of a job works. (It seems that overriding neutron-tempest-linuxbridge
> irrelevant-files works too).
>
> [7]
> https://github.com/openstack/neutron/blob/7e3d6a18fb928bcd303a44c1736d0d6ca9c7f0ab/.zuul.yaml#L140-L159
> [8]
> https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10516-L10530
> [9] https://review.openstack.org/#/c/537936/
>
> I don't see what is the difference between neutron-grenade and
> neutron-grenade-multinode jobs definitions from this perspective but
> it seems that the irrelevent-files attribute behaves  inconsistently
> in these two jobs. Could you please help me undestand how
> irrelevant-files in overriden jobs supposed to work?

These jobs only have the one invocation -- on the nova project -- and
are not added via a template.

Hopefully that explains the difference.

Basically, the irrelevant-files on at least one project-pipeline
invocation of a job have to match, as well as at least one definition of
the job.  So if both things have irrelevant-files, then it's effectively
a union of the two.

I used a tool to help verify some of the information in this message,
especially the bugs [5] and [6].  You can ask Zuul to output debug
information about its job selection if you're dealing with confusing
situations like this.  I went ahead and pushed a new patchset to your
test change to demonstrate how:

  https://review.openstack.org/537936

When it finishes running all the tests (in a few hours), it should
include in its report debug information about the decision-making
process for the jobs it ran.  It outputs similar information into the
debug logs; so that we don't have to wait for it to see what it looks
like here is that copy:

  http://paste.openstack.org/show/653729/

The relevant lines for [5] are:

2018-01-26 13:07:53,560 DEBUG zuul.layout: Pipeline variant <Job tempest-full branches: None source: openstack-infra/openstack-zuul-jobs/zuul.d/zuul-legacy-project-templates.yaml at master#126> matched <Change 0x7f2fed8883c8 537936,2>
2018-01-26 13:07:53,560 DEBUG zuul.layout: Pipeline variant <Job tempest-full branches: None source: openstack-infra/project-config/zuul.d/projects.yaml at master#10485> did not match <Change 0x7f2fed8883c8 537936,2>

Note the project-file-branch-line-number references are especially
helpful.

-Jim