Open Stack

Heya,

I started working on the Zuul Migration script - largely because we're 
starting to have more mappings of old job to new job than I can keep in 
my head and I wanted to be able to write them down as we make them.

https://review.openstack.org/#/q/topic:zuul-v3-migration

I doesn't do much yet - but I did add a zuul-v3 job that runs the 
migration on project-config and collects the results so we should be 
able to examine that as we work on it.

I've run in to a conceptual issue already that I'd like feedback on.

In dealing with project-templates, we have a bit of an issue. Take this 
snippet from layout.yaml:

project-templates:
   - name: loci-jobs
     check:
       - 'gate-{name}-ubuntu-xenial'
     gate:
       - 'gate-{name}-ubuntu-xenial'

We have two choices of how we can do transforms of things we don't 
explicitly map. We can expand them all and make completely equivilent 
jobs, so that we'd generate the following jobs:

- job:
     name: gate-loci-cinder

- job:
     name: gate-loci-glance

If we do that, then we can't really migrate the templates and would have 
to put expanded templates into the resulting project pipeline config, so 
we'd have:

- project:
     name: openstack/loci-cinder
     check:
       - gate-loci-cinder
     gate:
       - gate-loci-cinder

this is the safest - we KNOW that we can do this migration today and the 
jobs will all work as today - but the resulting config will be uglier. 
This is also a data loss as we'll lose ALL of our current templates.

We could also depend on the mapping construct and define a mapping for 
these loci jobs:

- old: gate-{name}-ubuntu-xenial
   name: test-loci

in which case we could make:

- job:
     name: test-loci

- project-template:
     name: loci-jobs
     check:
       - test-loci
     gate:
       - test-loci

- project:
     name: openstack/loci-cinder
     templates:
       - loci-jobs

But doing this will require a much more careful audit of our output - we 
cannot be sure that cases we haven't looked at will work and we 
essentially need to hand-verify every project-template to make sure the 
jobs it contains don't have edge-conditions we need to map.

As a third option - we could auto-expand {name} in project-templates to 
the name of the project template and generate jobs that way:

- job:
     name: gate-loci-jobs

- project-template:
     name: loci-jobs
     check:
       - gate-loci-jobs
     gate:
       - gate-loci-jobs

- project:
     name: openstack/loci-cinder
     templates:
       - loci-jobs

This is automatic and doesn't have data loss - but might be a little 
confusing. It also means for some of our templates we'll have duplicate 
jobs we don't need. {name}-tarball, for instance, shows up in many 
template. Now - we'll have a mapping for {name}-tarball - but for jobs 
that we don't have explicit mappings for and that do show up in multiple 
templates it might be weird.

Thoughts?