Open Stack

"Karpenko, Oleksandr (Nokia - DE/Ulm)" <oleksandr.karpenko at nokia.com>
writes:

> Hallo together,
>
> Sorry for bothering you but I am not able to find any zuul specific mailing list.
> First of all if there is some better place to ask questions, please
> let me know and I will do it there.

You've come to the right place.

> We are considering to use Zuul in a project split across many git
> repositories and we are in a proof of concept phase now. I have
> submitted for the review first problems we have found
> (https://review.openstack.org/#/c/423337/ ,
> https://review.openstack.org/#/c/424055/).

Thanks for those changes to the gearman-plugin for Jenkins.  I think the
first especially may be of use to a lot of people who have run into
permissions problems with newer Jenkins.

> Gerrit has a nice feature called 'Submit whole topic'. You can have
> reviews in several repositories within the same topic and gerrit
> allows to submit them together very close in time. When the first
> change is ready for Submit, 'Submit' button stays gray until all other
> reviews in the same topic are ready and then by pressing the button in
> any of those changes all of them are submitted.
>
> We have gate pipeline (DependentPipeline) in Zuul for Cross Project
> Testing which is triggered on code-review: 2 and on success does
> gerrit submit. The gate pipeline runs when the first change gets
> code-review: 2 and if all other changes have also code-review: 2, zuul
> submits all of them without testing the rest.
>
> Does Zuul have any support for 'Submit whole topic' gerrit feature? I
> am not able to find any.
>
> Does DependentPipeline have support for gerrit topic concept? It
> sounds at least reasonable to wait until all changes in the same topic
> get reviewed and then start DependentPipeline once (instead of 3 times
> in case of 2 repositories).
>
> Does our concept make sense at all or there are other ways to do it?
> We would like to reduce an effect of 'diamond dependency problem' by
> submitting related to each other changes close in time. For example
> when A depends on B and C. B and C depends on D. We would like to test
> A with related to each other changes in B and C (e.g. D integration)
> and submit changes in B and C very close in time. Because when B
> integrates D is much faster as C, A cannot be built until C is ready
> with D integration. This is not exactly what 'Depends-On:' is doing
> because changes in B and C does not depend on each other.
>
> Once more sorry for your time and looking forward for your answer.

Clint replied earlier with some good and correct information -- here is
a little more context and some thoughts about future development.

We implemented the Depens-On header feature in Zuul based on some
earlier work on cross-repo dependencies in Gerrit.  We have found the
'topic' to be useful in gerrit for far more things than just indicating
changes that should be merged together, and I personally am sad to see
it used for cross-repo dependencies as it robs Gerrit users of a useful
tool.

Regardless, you are correct that we only support one-way dependencies in
Zuul.  That is because we decided that, for the OpenStack project, we
wanted to avoid changing two repositories at once.  Many of our projects
are continuously deployed, and an OpenStack installation spans many
hosts (and also contains client-server relationships).  Therefore, we
wanted to ensure that developers changed each component in such a way
that it was backwards-compatible with other components.  This is
especially useful in the environment that I described, but also ends up
being good practice in more traditional application + library
situations.

However, I can see that some environments might prefer bidirectional
cross-repo dependencies.  Many of the building blocks needed to develop
that are already in zuul (we wrote the cross-repo-dependency code with
the idea that we might extend it to support this case in the future),
but there is still some work remaining to implement that.

If one were to implement it, I would recommend continuing to avoid using
the submit-whole-topic feature in Gerrit (as it is specific to Gerrit
and therefore not applicable to other systems with which we want to
interface Zuul in the future, and also, it is not necessary for this
work).  An implementation in Zuul should simply test both repos with the
state of the other, and then push the prepared repo states that it
tested up to Gerrit rather than 'submitting' the change using Gerrit.
This would allow Zuul to be certain that both merges will succeed (since
it has already performed them locally).

I think that we will end up implementing this eventually, but it is not
high on our list of priorities at the moment.  If you are interested in
working on this sooner, please let me know.

Finally, I would like to note that since you are in the exploration
stage that we are currently working heavily on version 3 of Zuul which
uses Ansible for running jobs rather than Jenkins.  We expect it to be
just as flexible terms of job content (if not more so), and while it's
not certain how much we will be able to automate this, we do expect to
have some tools for migrating some jenkins-job-builder configurations to
Ansible.  It's worth being aware of the direction we are heading as you
look into using Zuul.

Thanks,

Jim