Open Stack

On Sat, Jan 27, 2018 at 2:57 AM, James E. Blair <corvus at inaugust.com> wrote:
> Balázs Gibizer <balazs.gibizer at ericsson.com> writes:
>
>> Hi,
>>
>> I'm getting more and more confused how the zuul job hierarchy works or
>> is supposed to work.
>
> Hi!
>
> First, you (or others) may or may not have seen this already -- some of
> it didn't exist when we first rolled out v3, and some of it has changed
> -- but here are the relevant bits of the documentation that should help
> explain what's going on.  It helps to understand freezing:
>
>   https://docs.openstack.org/infra/zuul/user/config.html#job
>
> and matching:
>
>   https://docs.openstack.org/infra/zuul/user/config.html#matchers
>
>> First there was a bug in nova that some functional tests are not
>> triggered although the job (re-)definition in the nova part of the
>> project-config should not prevent it to run [1].
>>
>> There we figured out that irrelevant-files parameter of the jobs are
>> not something that can be overriden during re-definition or through
>> parent-child relationship. The base job openstack-tox-functional has
>> an irrelevant-files attribute that lists '^doc/.*$' as a path to be
>> ignored [2]. In the other hand the nova part of the project-config
>> tries to make this ignore less broad by adding only '^doc/source/.*$'
>> . This does not work as we expected and the job did not run on changes
>> that only affected ./doc/notification_samples path. We are fixing it
>> by defining our own functional job in nova tree [4].
>>
>> [1] https://bugs.launchpad.net/nova/+bug/1742962
>> [2]
>> https://github.com/openstack-infra/openstack-zuul-jobs/blob/1823e3ea20e6dfaf37786a6ff79c56cb786bf12c/zuul.d/jobs.yaml#L380
>> [3]
>> https://github.com/openstack-infra/project-config/blob/1145ab1293f5fa4d34c026856403c22b091e673c/zuul.d/projects.yaml#L10509
>> [4] https://review.openstack.org/#/c/533210/
>
> This is correct.  The issue here is that the irrelevant-files definition
> on openstack-tox-functional is too broad.  We need to be *extremely*
> careful applying matchers to jobs like that.  Generally I think that
> irrelevant-files should be reserved for the project-pipeline invocations
> only.  That's how they were effectively used in Zuul v2, after all.
>
> Essentially, when someone puts an irrelevant-files section on a job like
> that, they are saying "this job will never apply to these files, ever."
> That's clearly not correct in this case.
>
> So our solutions are to acknowledge that it's over-broad, and reduce or
> eliminate the list in [2] and expand it elsewhere (as in [3]).  Or we
> can say "we were generally correct, but nova is extra special so it
> needs its own job".  If that's the choice, then I think [4] is a fine
> solution.
>
>> Then I started looking into other jobs to see if we made similar
>> mistakes. I found two other examples in the nova related jobs where
>> redefining the irrelevant-files of a job caused problems. In these
>> examples nova tried to ignore more paths during the override than what
>> was originally ignored in the job definition but that did not work
>> [5][6].
>>
>> [5] https://bugs.launchpad.net/nova/+bug/1745405 (temptest-full)
>
> As noted in that bug, the tempest-full job is invoked on nova via this
> stanza:
>
> https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10674-L10688
>
> As expected, that did not match.  There is a second invocation of
> tempest-full on nova here:
>
> http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-project-templates.yaml#n126
>
> That has no irrelevant-files matches, and so matches everything.  If you
> drop the use of that template, it will work as expected.  Or, if you can
> say with some certainty that nova's irrelevant-files set is not
> over-broad, you could move the irrelevant-files from nova's invocation
> into the template, or even the job, and drop nova's individual
> invocation.
>
>> [6] https://bugs.launchpad.net/nova/+bug/1745431 (neutron-grenade)
>
> The same template invokes this job as well.
>
>> So far the problem seemed to be consistent (i.e. override does not
>> work). But then I looked into neutron-grenade-multinode. That job is
>> defined in neutron tree (like neutron-grenade) but nova also refers to
>> it in nova section of the project-config with different
>> irrelevant-files than their original definition. So I assumed that
>> this will lead to similar problem than in case of neutron-grenade, but
>> it doesn't.
>>
>> The neutron-grenade-multinode original definition [1] does not try to
>> ignore the 'nova/tests' path but the nova side of the definition in
>> the project config does try to ignore that path [8]. Interestingly a
>> patch in nova that only changes under the path: nova/tests/ does not
>> trigger the job [9]. So in this case overriding the irrelevant-files
>> of a job works. (It seems that overriding neutron-tempest-linuxbridge
>> irrelevant-files works too).
>>
>> [7]
>> https://github.com/openstack/neutron/blob/7e3d6a18fb928bcd303a44c1736d0d6ca9c7f0ab/.zuul.yaml#L140-L159
>> [8]
>> https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10516-L10530
>> [9] https://review.openstack.org/#/c/537936/
>>
>> I don't see what is the difference between neutron-grenade and
>> neutron-grenade-multinode jobs definitions from this perspective but
>> it seems that the irrelevent-files attribute behaves  inconsistently
>> in these two jobs. Could you please help me undestand how
>> irrelevant-files in overriden jobs supposed to work?
>
> These jobs only have the one invocation -- on the nova project -- and
> are not added via a template.
>
> Hopefully that explains the difference.
>
> Basically, the irrelevant-files on at least one project-pipeline
> invocation of a job have to match, as well as at least one definition of
> the job.  So if both things have irrelevant-files, then it's effectively
> a union of the two.

Thanks a lot for clarifying those bits. I thought irrelevant-files are
always overridden not append/union as they are regular expression or
list of regular expressions.

It is clear now that where irrelevant-files are present i both job
definition and project-pipeline invocation job list then it is union
of these two.
Is it same for inherited job case also or it is completely overridden?
I mean if base job has irrelevant-files and then inherited job from
this base job define irrelevant-files then, is it union or overridden.

One more question, as mentioned in doc that 'irrelevant-files' and
'files' should not be defined together in single job. what happen if
it is defined? as base job usually have 'irrelevant-files' and if any
inherited job defined 'files' then, is it error or unexpected
behaviour ?

Because answer to above query convey whether we should declare the
'irrelevant-files', 'files' in job definition or not ?

>
> I used a tool to help verify some of the information in this message,
> especially the bugs [5] and [6].  You can ask Zuul to output debug
> information about its job selection if you're dealing with confusing
> situations like this.  I went ahead and pushed a new patchset to your
> test change to demonstrate how:
>
>   https://review.openstack.org/537936
>
> When it finishes running all the tests (in a few hours), it should
> include in its report debug information about the decision-making
> process for the jobs it ran.  It outputs similar information into the
> debug logs; so that we don't have to wait for it to see what it looks
> like here is that copy:
>
>   http://paste.openstack.org/show/653729/
>
> The relevant lines for [5] are:
>
> 2018-01-26 13:07:53,560 DEBUG zuul.layout: Pipeline variant <Job tempest-full branches: None source: openstack-infra/openstack-zuul-jobs/zuul.d/zuul-legacy-project-templates.yaml at master#126> matched <Change 0x7f2fed8883c8 537936,2>
> 2018-01-26 13:07:53,560 DEBUG zuul.layout: Pipeline variant <Job tempest-full branches: None source: openstack-infra/project-config/zuul.d/projects.yaml at master#10485> did not match <Change 0x7f2fed8883c8 537936,2>
>
> Note the project-file-branch-line-number references are especially
> helpful.
>
> -Jim
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-gmann