Open Stack

Hi,

Jenkins-job-builder runs a job called "jenkins-job-builder-compare-xml"
which runs JJB master against openstack's JJB config, then runs it again
with the proposed change and compares the output.

Instead of success/failure, what it does is reports whether the output
changed or not.  If you follow the link, you can see exactly what
changed (it produces a diff).

Generally in JJB, by default we should expect that most changes do not
change the XML output.  Of course they can and should sometimes, which
is why the job doesn't vote.  In those cases, I think it's worth
highlighting either in the commit message or in the comments what you
would expect to change and why it's okay.

If the compare job says that the output changed, reviewers should ask
why it changed -- it may be an error.  Or if it was intentional, make
sure that the output of the compare job matches the intent of the
change.

Recently, a change was merged where JJB completely failed to run on the
production openstack-infra config.  Not only is this a problem for
infra, but it likely is a very legitimate bug in JJB itself.  The
problem was visible in the output of the compare XML job, which also
indicated that the XML output changed.  Hopefully in the future, we can
catch this in review.  If it would help catch people's attention, we can
change the message that Zuul leaves on the XML compare job.

-Jim