Open Stack

Greetings,

Over the last few weeks I've been helping the RDO project migrate away from
zuulv2 (jenkins) to zuulv3. Today all jobs have been migrated with the help of
the zuul-migrate script. We'll start deleting jenkins bits in the next few days.

I wanted to get down some things I've noticed in the process as feedback to
thirdparty CI operators. Hopefully this will help others.

Removal of zuul-cloner
----------------------

This by far was the largest issue we had in RDO project. The first thing it
meant, was the need for much more HDD space. We almost quadrupled the storage
quota needed to run zuulv3 properly because we no longer could zuul-cloner from
git.o.o.

Right now rdo is running 4 zuul-executors / 4 zuul-mergers, with the increase in
storage requirements this also meant we needed faster disks.  The previous
servers used under zuulv2 couldn't handle the IO now required, so we've had to
rebuild them backed with SSD. Previously they could be boot from volume to ceph.

Need for use-cached-repos
-------------------------

Today, use-cached-repos is only available to openstack-infra/project-config, we
should promote this into zuul-jobs to help reduce the amount of pressure on
zuul-executors when jobs start. In the case of 3pci, prepare-workspace role
isn't up to the task to sync everything at once.

The feedback here, is to some how allow the base job to be smart enough to work
if a project is found in /opt/git or not.  Today we have 2 different images in
rdo, 1 has the cache of upstream git.o.o and other doesn't.

Namespace projects with fqdn
----------------------------

This one is likely unique to rdoproject, but because we have 2 connection to
different gerrit systems, review.rdoproject.org and git.openstack.org, we
actually have duplicate project names. For example:

  openstack/tripleo-common

which means, for zuul we have to write projects as:

  project:
    name: git.openstack.org/openstack/tripleo-common

  project:
    name: review.openstack.org/openstack/tripleo-common

There are legacy reasons for this, and we plan on cleaning review.r.o, however
because of this duplication we cannot use upstream jobs right now. My initial
thought would be to update jobs, in this case devstack to use the following for
required-projects:

  required-projects:
    - git.openstack.org/openstack-dev/devstack
    - git.openstack.org/openstack/tripleo-common

and propose the patch upstream.  Again, this is likely specific to rdoproject,
but something right now that blocks them on loading jobs from zuul.o.o.

I do have some other suggestions, but they are more specific to zuul. I could
post them here as a follow up or on zuul ML.

I am happy I was able to help in the original migration of the openstack
projects from jenkins to zuulv3, it did help a lot when I was debugging zuul
failures. But over all rdo project didn't have any major issues with job content.

Thanks,
Paul