On Thu, 2021-02-04 at 22:54 +0000, Jeremy Stanley wrote:
On 2021-02-04 22:29:35 +0200 (+0200), Dmitriy Rabotyagov wrote:
For OSA huge issue is how zuul clones required-projects. Just this single action takes for us from 6 to 10 minutes. [...]
I'd be curious to see some examples of this. Zuul doesn't clone required-projects, but it does push new commits from a cache on the executor to a cache on the node.
right originally that was don by https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/prepare-workspace... as part of the base job https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/pre.y... that just sysncs the deltas using git between the git repo prepared on the zuul executor with those precached in the images. that will however only updte teh repos that are needed by your job via required-projects. it ensure that you are actully testing what you think you are testing and that you never need to cloue any of the repos you are test since they will be prepared for you. this should be much faster then clonneing or pulling the repos yourself in the job and more imporantly it avoid netwokring issue that can happen if you try to clone in the job from gerrit. its also what makes depends on work which is tricky if you dont leave zuul prepare the repos for you.
The executor side caches are updated continually as new builds are scheduled, and the caches on the nodes are refreshed every time the images from which they're booted are assembled (typically daily unless something has temporarily broken our ability to rebuild a particular image). So on average, the executor is pushing only 12 hours worth of new commits for each required project. I don't recall if it performs those push operations in parallel, but I suppose that's something we could look into.
its not parallel if im reading this write but typically you will not need to pull alot of repos. https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/prepare-workspace... as in its maybe in the low 10s for typical full tempest jobs. OSA is slightly pathalogical in howe it use require porjects https://opendev.org/openstack/openstack-ansible/src/branch/master/zuul.d/job... its pulling in a large propoation of the openstack repos. its not suprising its slower then we typeicaly would expect but it shoudl be fater then if you actully clonned them without useing the cache in the image and executor. doing the clone in parallel would help in this case but it might also make sense to reasses how osa stuctures its jobs for example osa support both source and non souce installs correct. the non souce installs dont need the openstack pojects just the osa repos since it will be using the binary packages. so if you had a second intermeitady job of the souce install the the openstack compoenta repos listed you could skip updating 50 repos in your binary jobs (im assumning the _distro_ jobs are binary by the way.) currenlty its updating 105 for every job that is based on openstack-ansible-deploy-aio