Open Stack

We have to be wary about how projects tend to try and detect if ZUUL_*
variables [1] and zuul-cloner [2] are available in different ways and
do appropriate things based on the result.

It's hard to find every instance of that in codesearch, though... so I
wonder if there is a way to detect if any shims or deprecated things
are used.
Perhaps have the shim print/log a unique signature (could just be a
echo of "You're using a deprecated thing: <thing>") and then we can
identify jobs using those shims in logstash and fix them over a period
of time.

And even then, there might be the problem of third party CI. I don't
know if there are many instances of Zuul v2 used for third party in
the wild, the only one I know of is review.rdoproject.org [3].
We'll upgrade review.rdoproject.org eventually for sure but there's
probably nothing stopping existing v2 users from staying on that
version for ever.

Would our migration to v3 impact non-zuul third party CI users ?

[1]: http://codesearch.openstack.org/?q=ZUUL_&i=nope&files=&repos=
[2]: http://codesearch.openstack.org/?q=.*(which%7Cif.*).*cloner.*&i=nope&files=&repos=
[3]: https://review.rdoproject.org/zuul/

David Moreau Simard
Senior Software Engineer | OpenStack RDO

dmsimard = [irc, github, twitter]


On Thu, Aug 31, 2017 at 6:44 PM, James E. Blair <corvus at inaugust.com> wrote:
> Hi,
>
> Something we need to be aware of for the Zuul v3 migration is the use of
> the zuul-cloner tool.  In many of the more "standardized" jobs, this
> will be taken care of either by the fact that we have reworked the jobs
> into native v3 format (eg unittests), or the migration script will map
> existing JJB builders (zuul-git-prep) into v3 equivalents.  However,
> there are many jobs which perform their own zuul-cloner actions, some
> defined in project-config, and others scattered throughout project
> repos.
>
> In those cases, it would take considerable time to track down all the
> uses and put a migration plan in place.  Instead, I propose a
> compatibility shim.
>
> We can create a new script which takes the same arguments as
> zuul-cloner.  It will ignore most of the arguments except the repos and
> the destination which it will use to copy the specified repos from
> ~zuul/src/git.openstack.org/... to the destination.
>
> When jobs run this script, they will get the Zuul v3 prepared repos, and
> they will be placed in the locations that the job expects.  If it can
> support the clonemap file, that's even better.
>
> If jobs provide arguments to zuul-cloner which alter its behavior about
> which branches are checked out, they may fail with this plan.  It will
> be easy to correct this by altering the Zuul v3 job configuration to
> specify override branches, however, this is likely a situation which
> will require human intervention to fix.  Ideally, there won't be very
> many of these cases.
>
> Additionally, any of the repos used by zuul-cloner in a job will need to
> be specified as required-projects in Zuul.  Determining this
> automatically is hard, but we can handle many cases with heuristics.
> For example, a substantial number of these jobs are for neutron
> plugins.  We can easily tell the migration script to add 'neutron' as a
> required-project for all such jobs.
>
> Zuul-cloner is baked into our images.  In order to allow us to easily
> move between Zuul v2 and v3, I propose we have the Zuul v3 base job
> automatically replace the zuul-cloner script with the one proposed here,
> so we can continue to use the same images for Zuul v2 and v3 during the
> transition.
>
> Ultimately, it would be useful to have this in Zuul v3 for other users.
> It probably makes the most sense to add this to the feature/zuulv3
> branch as the zuul-cloner binary.  We can release it as a deprecated
> migration tool.  The base job can just wget it from git.o.o for
> simplicity.  To facilitate that, we may want to ensure the script is
> self-contained.  After we fully migrate to v3, we can bake it into the
> image again.
>
> -Jim
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra