Open Stack

On Fri, Jan 26, 2018 at 6:57 PM, James E. Blair <corvus at inaugust.com> 
wrote:
> Balázs Gibizer <balazs.gibizer at ericsson.com> writes:
> 
>>  Hi,
>> 
>>  I'm getting more and more confused how the zuul job hierarchy works 
>> or
>>  is supposed to work.
> 
> Hi!
> 
> First, you (or others) may or may not have seen this already -- some 
> of
> it didn't exist when we first rolled out v3, and some of it has 
> changed
> -- but here are the relevant bits of the documentation that should 
> help
> explain what's going on.  It helps to understand freezing:
> 
>   https://docs.openstack.org/infra/zuul/user/config.html#job
> 
> and matching:
> 
>   https://docs.openstack.org/infra/zuul/user/config.html#matchers

Thanks for the doc references they are really helpful.

> 
>>  First there was a bug in nova that some functional tests are not
>>  triggered although the job (re-)definition in the nova part of the
>>  project-config should not prevent it to run [1].
>> 
>>  There we figured out that irrelevant-files parameter of the jobs are
>>  not something that can be overriden during re-definition or through
>>  parent-child relationship. The base job openstack-tox-functional has
>>  an irrelevant-files attribute that lists '^doc/.*$' as a path to be
>>  ignored [2]. In the other hand the nova part of the project-config
>>  tries to make this ignore less broad by adding only 
>> '^doc/source/.*$'
>>  . This does not work as we expected and the job did not run on 
>> changes
>>  that only affected ./doc/notification_samples path. We are fixing it
>>  by defining our own functional job in nova tree [4].
>> 
>>  [1] https://bugs.launchpad.net/nova/+bug/1742962
>>  [2]
>>  
>> https://github.com/openstack-infra/openstack-zuul-jobs/blob/1823e3ea20e6dfaf37786a6ff79c56cb786bf12c/zuul.d/jobs.yaml#L380
>>  [3]
>>  
>> https://github.com/openstack-infra/project-config/blob/1145ab1293f5fa4d34c026856403c22b091e673c/zuul.d/projects.yaml#L10509
>>  [4] https://review.openstack.org/#/c/533210/
> 
> This is correct.  The issue here is that the irrelevant-files 
> definition
> on openstack-tox-functional is too broad.  We need to be *extremely*
> careful applying matchers to jobs like that.  Generally I think that
> irrelevant-files should be reserved for the project-pipeline 
> invocations
> only.  That's how they were effectively used in Zuul v2, after all.
> 
> Essentially, when someone puts an irrelevant-files section on a job 
> like
> that, they are saying "this job will never apply to these files, 
> ever."
> That's clearly not correct in this case.
> 
> So our solutions are to acknowledge that it's over-broad, and reduce 
> or
> eliminate the list in [2] and expand it elsewhere (as in [3]).  Or we
> can say "we were generally correct, but nova is extra special so it
> needs its own job".  If that's the choice, then I think [4] is a fine
> solution.

The [4] just get merged this morning so I think that is OK for us now.

> 
>>  Then I started looking into other jobs to see if we made similar
>>  mistakes. I found two other examples in the nova related jobs where
>>  redefining the irrelevant-files of a job caused problems. In these
>>  examples nova tried to ignore more paths during the override than 
>> what
>>  was originally ignored in the job definition but that did not work
>>  [5][6].
>> 
>>  [5] https://bugs.launchpad.net/nova/+bug/1745405 (temptest-full)
> 
> As noted in that bug, the tempest-full job is invoked on nova via this
> stanza:
> 
> https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10674-L10688
> 
> As expected, that did not match.  There is a second invocation of
> tempest-full on nova here:
> 
> http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-project-templates.yaml#n126
> 
> That has no irrelevant-files matches, and so matches everything.  If 
> you
> drop the use of that template, it will work as expected.  Or, if you 
> can
> say with some certainty that nova's irrelevant-files set is not
> over-broad, you could move the irrelevant-files from nova's invocation
> into the template, or even the job, and drop nova's individual
> invocation.

Thanks for the explanation, it is much clearer now. With this info I 
think I was able to propose a patcha that fixes the two bugs: 
https://review.openstack.org/#/c/538908/

> 
>>  [6] https://bugs.launchpad.net/nova/+bug/1745431 (neutron-grenade)
> 
> The same template invokes this job as well.
> 
>>  So far the problem seemed to be consistent (i.e. override does not
>>  work). But then I looked into neutron-grenade-multinode. That job is
>>  defined in neutron tree (like neutron-grenade) but nova also refers 
>> to
>>  it in nova section of the project-config with different
>>  irrelevant-files than their original definition. So I assumed that
>>  this will lead to similar problem than in case of neutron-grenade, 
>> but
>>  it doesn't.
>> 
>>  The neutron-grenade-multinode original definition [1] does not try 
>> to
>>  ignore the 'nova/tests' path but the nova side of the definition in
>>  the project config does try to ignore that path [8]. Interestingly a
>>  patch in nova that only changes under the path: nova/tests/ does not
>>  trigger the job [9]. So in this case overriding the irrelevant-files
>>  of a job works. (It seems that overriding 
>> neutron-tempest-linuxbridge
>>  irrelevant-files works too).
>> 
>>  [7]
>>  
>> https://github.com/openstack/neutron/blob/7e3d6a18fb928bcd303a44c1736d0d6ca9c7f0ab/.zuul.yaml#L140-L159
>>  [8]
>>  
>> https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10516-L10530
>>  [9] https://review.openstack.org/#/c/537936/
>> 
>>  I don't see what is the difference between neutron-grenade and
>>  neutron-grenade-multinode jobs definitions from this perspective but
>>  it seems that the irrelevent-files attribute behaves  inconsistently
>>  in these two jobs. Could you please help me undestand how
>>  irrelevant-files in overriden jobs supposed to work?
> 
> These jobs only have the one invocation -- on the nova project -- and
> are not added via a template.
> 
> Hopefully that explains the difference.

OK, now I think I see the difference. So far I mixed  the definition 
and the invocation of a job in my head.  Thanks for the explanation. 
Your mail really helped me understand the whole situation.

cheers,
gibi

> 
> Basically, the irrelevant-files on at least one project-pipeline
> invocation of a job have to match, as well as at least one definition 
> of
> the job.  So if both things have irrelevant-files, then it's 
> effectively
> a union of the two.
> 
> I used a tool to help verify some of the information in this message,
> especially the bugs [5] and [6].  You can ask Zuul to output debug
> information about its job selection if you're dealing with confusing
> situations like this.  I went ahead and pushed a new patchset to your
> test change to demonstrate how:
> 
>   https://review.openstack.org/537936
> 
> When it finishes running all the tests (in a few hours), it should
> include in its report debug information about the decision-making
> process for the jobs it ran.  It outputs similar information into the
> debug logs; so that we don't have to wait for it to see what it looks
> like here is that copy:
> 
>   http://paste.openstack.org/show/653729/
> 
> The relevant lines for [5] are:
> 
> 2018-01-26 13:07:53,560 DEBUG zuul.layout: Pipeline variant <Job 
> tempest-full branches: None source: 
> openstack-infra/openstack-zuul-jobs/zuul.d/zuul-legacy-project-templates.yaml at master#126> 
> matched <Change 0x7f2fed8883c8 537936,2>
> 2018-01-26 13:07:53,560 DEBUG zuul.layout: Pipeline variant <Job 
> tempest-full branches: None source: 
> openstack-infra/project-config/zuul.d/projects.yaml at master#10485> did 
> not match <Change 0x7f2fed8883c8 537936,2>
> 
> Note the project-file-branch-line-number references are especially
> helpful.
> 
> -Jim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180129/ab37dd13/attachment.html>