Open Stack

On Thu, 2021-02-04 at 22:54 +0000, Jeremy Stanley wrote:
> On 2021-02-04 22:29:35 +0200 (+0200), Dmitriy Rabotyagov wrote:
> > For OSA huge issue is how zuul clones required-projects. Just this
> > single action takes for us from 6 to 10 minutes.
> [...]
> 
> I'd be curious to see some examples of this. Zuul doesn't clone
> required-projects, but it does push new commits from a cache on the
> executor to a cache on the node.
> 
right originally that was don by 
https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/prepare-workspace-git
as part of the base job https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/pre.yaml#L33
that just sysncs the deltas using git between the git repo prepared on the zuul executor
with those precached in the images.

that will however only updte teh repos that are needed by your job via required-projects.
it ensure that you are actully testing what you think you are testing and that you
never need to cloue any of the repos you are test since they will be prepared for you.

this should be much faster then clonneing or pulling the repos yourself in the job and
more imporantly it avoid netwokring issue that can happen if you try to clone in the job
from gerrit. its also what makes depends on work which is tricky if you dont leave zuul
prepare the repos for you.

>  The executor side caches are
> updated continually as new builds are scheduled, and the caches on
> the nodes are refreshed every time the images from which they're
> booted are assembled (typically daily unless something has
> temporarily broken our ability to rebuild a particular image). So on
> average, the executor is pushing only 12 hours worth of new commits
> for each required project. I don't recall if it performs those push
> operations in parallel, but I suppose that's something we could look
> into.

its not parallel if im reading this write but typically you will not need to pull alot of repos.
https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/prepare-workspace-git/tasks/main.yaml#L11-L43
as in its maybe in the low 10s for typical full tempest jobs.

OSA is slightly pathalogical in howe it use require porjects

https://opendev.org/openstack/openstack-ansible/src/branch/master/zuul.d/jobs.yaml#L23-L132

its pulling in a large propoation of the openstack repos.
its not suprising its slower then we typeicaly would expect but it shoudl be fater then if you actully clonned
them without useing the cache in the image and executor.

doing the clone in parallel would help in this case but it might also make sense to reasses how osa stuctures its jobs
for example osa support both source and non souce installs correct. the non souce installs dont need the openstack pojects
just the osa repos since it will be using the binary packages.

so if you had a second intermeitady job of the souce install the the openstack compoenta repos listed you could skip 
updating 50 repos in your binary jobs (im assumning the _distro_ jobs are binary by the way.)
currenlty its updating 105 for every job that is based on openstack-ansible-deploy-aio