Open Stack

On Fri, May 24, 2019 at 7:37 PM Clark Boylan <cboylan at sapwetik.org> wrote:

> On Fri, May 24, 2019, at 10:23 AM, Alfredo Moralejo Alonso wrote:
> > Hi,
> >
> > Since last hours puppet-openstack-integrating jobs in RDO are failing
> > to deploy puppet modules from
> > https://opendev.org/openstack/puppet-<modules> in CentOS7.
> >
> > The sympthom is jobs get stuck randomly when installing modules using
> > r10k. Investigating it, I've been able to reproduce the issue running:
> >
> > git clone --mirror https://github.com/openstack/puppet-<module>
> >
> > About 50% of the times, commands get stuck for any project in
> > opendev.org/openstack. I've checked that it works fine if i don't use
> > "--mirror".
> >
> > As it seems to work fine with fedora, I've compiled and installed
> > latest version of git from upstream and in a CentOS7 and then, git
> > clone --mirror works fine, but in this case r10k also gets stuck when
> > running:
> >
> > git --git-dir
> > /home/centos/.r10k/git/https---opendev.org-openstack-puppet-keystone
> > fetch origin --prune
> >
> > In this case, it only happens with puppet-keystone.
> >
> > Note that it always works fine using github.com/openstack mirrors
> > instead of opendev.org so the problem seems to be some combination of
> > git repos from opendev with CentOS7 git. Has there been any change that
> > may be affecting git repos behavior somehow?
>
> Yesterday we replicated all of refs/changes and refs/notes from Gerrit to
> Gitea. We had these replicated on the old Cgit farm, but gitea didn't
> support this replication until wednesday when we upgraded Gitea. The
> manpage for git clone --mirror says it will "map all refs (including
> remote-tracking branches, notes etc.)" so this is possibly related.
>
> When you say it gets stuck randomly does that mean the git process never
> completes the clone? or it stops with failure? In either case stracing the
> clone process might help to figure out if it is these new refs that it is
> having problems with.
>
>
When fails, it never completes, example output with GIT_TRACE enabled:

$ git  clone --mirror https://opendev.org/openstack/puppet-neutron
trace: built-in: git 'clone' '--mirror' '
https://opendev.org/openstack/puppet-neutron'
Cloning into bare repository 'puppet-neutron.git'...
trace: run_command: 'git-remote-https' 'origin' '
https://opendev.org/openstack/puppet-neutron'
trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack'
'--thin' 'https://opendev.org/openstack/puppet-neutron/'
trace: exec: 'git' 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack'
'--thin' 'https://opendev.org/openstack/puppet-neutron/'
trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack'
'--thin' 'https://opendev.org/openstack/puppet-neutron/'

It always get stuck in:

trace: built-in: git 'fetch-pack' ...

I have a 2MB strace output, but it's probably more useful for you to just
replicate it in a centos:latest container?, simply:

"
docker run -it docker.io/centos:latest /bin/bash

yum install -y git strace

git clone --mirror https://opendev.org/openstack/puppet-nova
"
you can run the git clone several times if it finishes fine.




> Clark
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20190524/0bade7dc/attachment-0001.html>