<div id="geary-body"><div><br></div></div><div id="geary-quote">On Fri, Jan 26, 2018 at 6:57 PM, James E. Blair <corvus@inaugust.com> wrote:<br><blockquote type="cite"><div class="plaintext" style="white-space: pre-wrap;">Balázs Gibizer <<a href="mailto:balazs.gibizer@ericsson.com">balazs.gibizer@ericsson.com</a>> writes:
<blockquote> Hi,
I'm getting more and more confused how the zuul job hierarchy works or
is supposed to work.
</blockquote>
Hi!
First, you (or others) may or may not have seen this already -- some of
it didn't exist when we first rolled out v3, and some of it has changed
-- but here are the relevant bits of the documentation that should help
explain what's going on. It helps to understand freezing:
<a href="https://docs.openstack.org/infra/zuul/user/config.html#job">https://docs.openstack.org/infra/zuul/user/config.html#job</a>
and matching:
<a href="https://docs.openstack.org/infra/zuul/user/config.html#matchers">https://docs.openstack.org/infra/zuul/user/config.html#matchers</a></div></blockquote><span style="white-space: pre-wrap;"><div><br></div><div>Thanks for the doc references they are really helpful. </div><br></span><blockquote type="cite"><div class="plaintext" style="white-space: pre-wrap;">
<blockquote> First there was a bug in nova that some functional tests are not
triggered although the job (re-)definition in the nova part of the
project-config should not prevent it to run [1].
There we figured out that irrelevant-files parameter of the jobs are
not something that can be overriden during re-definition or through
parent-child relationship. The base job openstack-tox-functional has
an irrelevant-files attribute that lists '^doc/.*$' as a path to be
ignored [2]. In the other hand the nova part of the project-config
tries to make this ignore less broad by adding only '^doc/source/.*$'
. This does not work as we expected and the job did not run on changes
that only affected ./doc/notification_samples path. We are fixing it
by defining our own functional job in nova tree [4].
[1] <a href="https://bugs.launchpad.net/nova/+bug/1742962">https://bugs.launchpad.net/nova/+bug/1742962</a>
[2]
<a href="https://github.com/openstack-infra/openstack-zuul-jobs/blob/1823e3ea20e6dfaf37786a6ff79c56cb786bf12c/zuul.d/jobs.yaml#L380">https://github.com/openstack-infra/openstack-zuul-jobs/blob/1823e3ea20e6dfaf37786a6ff79c56cb786bf12c/zuul.d/jobs.yaml#L380</a>
[3]
<a href="https://github.com/openstack-infra/project-config/blob/1145ab1293f5fa4d34c026856403c22b091e673c/zuul.d/projects.yaml#L10509">https://github.com/openstack-infra/project-config/blob/1145ab1293f5fa4d34c026856403c22b091e673c/zuul.d/projects.yaml#L10509</a>
[4] <a href="https://review.openstack.org/#/c/533210/">https://review.openstack.org/#/c/533210/</a>
</blockquote>
This is correct. The issue here is that the irrelevant-files definition
on openstack-tox-functional is too broad. We need to be *extremely*
careful applying matchers to jobs like that. Generally I think that
irrelevant-files should be reserved for the project-pipeline invocations
only. That's how they were effectively used in Zuul v2, after all.
Essentially, when someone puts an irrelevant-files section on a job like
that, they are saying "this job will never apply to these files, ever."
That's clearly not correct in this case.
So our solutions are to acknowledge that it's over-broad, and reduce or
eliminate the list in [2] and expand it elsewhere (as in [3]). Or we
can say "we were generally correct, but nova is extra special so it
needs its own job". If that's the choice, then I think [4] is a fine
solution.</div></blockquote><span style="white-space: pre-wrap;"><br><div><span style="white-space: pre-wrap;">The [4] just get merged this morning so I think that is OK for us now.</span></div><br></span><blockquote type="cite"><div class="plaintext" style="white-space: pre-wrap;">
<blockquote> Then I started looking into other jobs to see if we made similar
mistakes. I found two other examples in the nova related jobs where
redefining the irrelevant-files of a job caused problems. In these
examples nova tried to ignore more paths during the override than what
was originally ignored in the job definition but that did not work
[5][6].
[5] <a href="https://bugs.launchpad.net/nova/+bug/1745405">https://bugs.launchpad.net/nova/+bug/1745405</a> (temptest-full)
</blockquote>
As noted in that bug, the tempest-full job is invoked on nova via this
stanza:
<a href="https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10674-L10688">https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10674-L10688</a>
As expected, that did not match. There is a second invocation of
tempest-full on nova here:
<a href="http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-project-templates.yaml#n126">http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-project-templates.yaml#n126</a>
That has no irrelevant-files matches, and so matches everything. If you
drop the use of that template, it will work as expected. Or, if you can
say with some certainty that nova's irrelevant-files set is not
over-broad, you could move the irrelevant-files from nova's invocation
into the template, or even the job, and drop nova's individual
invocation.</div></blockquote><div style="white-space: pre-wrap;"><span style="white-space: pre-wrap;"><br></span></div><div><span style="white-space: pre-wrap;">Thanks for the explanation, it is much clearer now. With this info I think I was able to propose a patcha that fixes the two bugs: https://review.openstack.org/#/c/538908/ </span></div><br><blockquote type="cite"><div class="plaintext" style="white-space: pre-wrap;">
<blockquote> [6] <a href="https://bugs.launchpad.net/nova/+bug/1745431">https://bugs.launchpad.net/nova/+bug/1745431</a> (neutron-grenade)
</blockquote>
The same template invokes this job as well.
<blockquote> So far the problem seemed to be consistent (i.e. override does not
work). But then I looked into neutron-grenade-multinode. That job is
defined in neutron tree (like neutron-grenade) but nova also refers to
it in nova section of the project-config with different
irrelevant-files than their original definition. So I assumed that
this will lead to similar problem than in case of neutron-grenade, but
it doesn't.
The neutron-grenade-multinode original definition [1] does not try to
ignore the 'nova/tests' path but the nova side of the definition in
the project config does try to ignore that path [8]. Interestingly a
patch in nova that only changes under the path: nova/tests/ does not
trigger the job [9]. So in this case overriding the irrelevant-files
of a job works. (It seems that overriding neutron-tempest-linuxbridge
irrelevant-files works too).
[7]
<a href="https://github.com/openstack/neutron/blob/7e3d6a18fb928bcd303a44c1736d0d6ca9c7f0ab/.zuul.yaml#L140-L159">https://github.com/openstack/neutron/blob/7e3d6a18fb928bcd303a44c1736d0d6ca9c7f0ab/.zuul.yaml#L140-L159</a>
[8]
<a href="https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10516-L10530">https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10516-L10530</a>
[9] <a href="https://review.openstack.org/#/c/537936/">https://review.openstack.org/#/c/537936/</a>
I don't see what is the difference between neutron-grenade and
neutron-grenade-multinode jobs definitions from this perspective but
it seems that the irrelevent-files attribute behaves inconsistently
in these two jobs. Could you please help me undestand how
irrelevant-files in overriden jobs supposed to work?
</blockquote>
These jobs only have the one invocation -- on the nova project -- and
are not added via a template.
Hopefully that explains the difference.</div></blockquote><span style="white-space: pre-wrap;"><div><span style="white-space: pre-wrap;"><br></span></div><div><span style="white-space: pre-wrap;">OK, now I think I see the difference. So far I mixed the definition and the invocation of a job in my head. Thanks for the explanation. Your mail really helped me understand the whole situation.</span></div><div><span style="white-space: pre-wrap;"><br></span></div><div><span style="white-space: pre-wrap;">cheers,</span></div><div><span style="white-space: pre-wrap;">gibi</span></div><br></span><blockquote type="cite"><div class="plaintext" style="white-space: pre-wrap;">
Basically, the irrelevant-files on at least one project-pipeline
invocation of a job have to match, as well as at least one definition of
the job. So if both things have irrelevant-files, then it's effectively
a union of the two.
I used a tool to help verify some of the information in this message,
especially the bugs [5] and [6]. You can ask Zuul to output debug
information about its job selection if you're dealing with confusing
situations like this. I went ahead and pushed a new patchset to your
test change to demonstrate how:
<a href="https://review.openstack.org/537936">https://review.openstack.org/537936</a>
When it finishes running all the tests (in a few hours), it should
include in its report debug information about the decision-making
process for the jobs it ran. It outputs similar information into the
debug logs; so that we don't have to wait for it to see what it looks
like here is that copy:
<a href="http://paste.openstack.org/show/653729/">http://paste.openstack.org/show/653729/</a>
The relevant lines for [5] are:
2018-01-26 13:07:53,560 DEBUG zuul.layout: Pipeline variant <Job tempest-full branches: None source: openstack-infra/openstack-zuul-jobs/zuul.d/zuul-legacy-project-templates.yaml@master#126> matched <Change 0x7f2fed8883c8 537936,2>
2018-01-26 13:07:53,560 DEBUG zuul.layout: Pipeline variant <Job tempest-full branches: None source: openstack-infra/project-config/zuul.d/projects.yaml@master#10485> did not match <Change 0x7f2fed8883c8 537936,2>
Note the project-file-branch-line-number references are especially
helpful.
-Jim
</div></blockquote></div>