Open Stack

On Tue, Aug 8, 2017, at 07:57 AM, Monty Taylor wrote:
> Heya,
> 
> I started working on the Zuul Migration script - largely because we're 
> starting to have more mappings of old job to new job than I can keep in 
> my head and I wanted to be able to write them down as we make them.
> 
> https://review.openstack.org/#/q/topic:zuul-v3-migration
> 
> I doesn't do much yet - but I did add a zuul-v3 job that runs the 
> migration on project-config and collects the results so we should be 
> able to examine that as we work on it.
> 
> I've run in to a conceptual issue already that I'd like feedback on.
> 
> In dealing with project-templates, we have a bit of an issue. Take this 
> snippet from layout.yaml:
> 
> project-templates:
>    - name: loci-jobs
>      check:
>        - 'gate-{name}-ubuntu-xenial'
>      gate:
>        - 'gate-{name}-ubuntu-xenial'
> 
> We have two choices of how we can do transforms of things we don't 
> explicitly map. We can expand them all and make completely equivilent 
> jobs, so that we'd generate the following jobs:
> 
> - job:
>      name: gate-loci-cinder
> 
> - job:
>      name: gate-loci-glance
> 
> If we do that, then we can't really migrate the templates and would have 
> to put expanded templates into the resulting project pipeline config, so 
> we'd have:
> 
> - project:
>      name: openstack/loci-cinder
>      check:
>        - gate-loci-cinder
>      gate:
>        - gate-loci-cinder
> 
> this is the safest - we KNOW that we can do this migration today and the 
> jobs will all work as today - but the resulting config will be uglier. 
> This is also a data loss as we'll lose ALL of our current templates.
> 
> We could also depend on the mapping construct and define a mapping for 
> these loci jobs:
> 
> - old: gate-{name}-ubuntu-xenial
>    name: test-loci
> 
> in which case we could make:
> 
> - job:
>      name: test-loci
> 
> - project-template:
>      name: loci-jobs
>      check:
>        - test-loci
>      gate:
>        - test-loci
> 
> - project:
>      name: openstack/loci-cinder
>      templates:
>        - loci-jobs
> 
> But doing this will require a much more careful audit of our output - we 
> cannot be sure that cases we haven't looked at will work and we 
> essentially need to hand-verify every project-template to make sure the 
> jobs it contains don't have edge-conditions we need to map.
> 
> As a third option - we could auto-expand {name} in project-templates to 
> the name of the project template and generate jobs that way:
> 
> - job:
>      name: gate-loci-jobs
> 
> - project-template:
>      name: loci-jobs
>      check:
>        - gate-loci-jobs
>      gate:
>        - gate-loci-jobs
> 
> - project:
>      name: openstack/loci-cinder
>      templates:
>        - loci-jobs
> 
> This is automatic and doesn't have data loss - but might be a little 
> confusing. It also means for some of our templates we'll have duplicate 
> jobs we don't need. {name}-tarball, for instance, shows up in many 
> template. Now - we'll have a mapping for {name}-tarball - but for jobs 
> that we don't have explicit mappings for and that do show up in multiple 
> templates it might be weird.
> 
> Thoughts?

For the "core" projects we have a fairly consistent set of jobs applied
to them and they would likely survive the second option with few
problems. However, all the other stuff out there there is a ton of
variety and it is hard or even impossible to infer job behavior based on
template names. Unfortunately, these are also the projects most likely
to not notice immediately when they have problems and to have the most
difficultly updating things on their own.

Another point to consider is jobs like
{trigger}-tripleo-ci-{jobname}{suffix} and
{pipeline}-kolla-dsvm-{action}-{distro}-{type}-{node}{suffix} where so
much more than just a project name is encoded into a job name. You can't
just smash that down to a single dimension sanely. For this reason it
might be worthwile doing option 1 by default then we can whitelist the
core set of jobs against the integrated gate to get clean config there.
Then we can show everyone else that cleaner config for integrated gate
in order to get them started porting their things in the nicer way.

Clark