<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
On 04/03/15 14:29, liuxinguo wrote:<br>
<blockquote
cite="mid:E97AE00C7CAEC3489EF8B5A429A8B2842DA764FA@szxeml504-mbx.china.huawei.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 12 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"MS Mincho";
panose-1:2 2 6 9 4 2 5 8 3 4;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"\@MS Mincho";
panose-1:2 2 6 9 4 2 5 8 3 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.def
{mso-style-name:def;}
.MsoChpDefault
{mso-style-type:export-only;}
/* Page Definitions */
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1877428236;
mso-list-template-ids:1680490112;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">The network of our CI is
poor so sometime the zuul will get stuck when fetching patch
from gerrit. When running “ps -ef|grep zuul” I find it get
stuck like this:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">root@3rd-ci-master:/var/log/zuul#
ps -ef|grep zuul<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">root 698 9074 0
16:48 pts/0 00:00:00 grep --color=auto zuul<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">zuul 31730 1 2
03:01 ? 00:16:37 /usr/bin/python
/usr/local/bin/zuul-server<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">zuul 31739 1 0
03:01 ? 00:00:00 /usr/bin/python
/usr/local/bin/zuul-merger<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">zuul 31748 31730 0
03:01 ? 00:00:00 /usr/bin/python
/usr/local/bin/zuul-server<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">zuul 32008 31739 0
05:08 ? 00:00:00 git remote update origin<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">zuul 32009 32008 0
05:08 ? 00:00:00 git fetch --multiple origin<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">zuul 32010 32009 0
05:08 ? 00:00:00 git fetch --append origin<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">zuul 32011 32010 0
05:08 ? 00:00:00 /bin/bash
/var/lib/zuul/git/.ssh_wrapper -p 29418
<a moz-do-not-send="true"
href="mailto:huawei-volume-ci@review.openstack.org">huawei-volume-ci@review.openstack.org</a>
git-upload-pack '/openstack/cinder'<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">zuul 32012 32011 0
05:08 ? 00:00:00 ssh -i /var/lib/zuul/ssh/id_rsa -p
29418
<a moz-do-not-send="true"
href="mailto:huawei-volume-ci@review.openstack.org">huawei-volume-ci@review.openstack.org</a>
git-upload-pack '/openstack/cinder'</span><span
style="font-family:"MS Mincho";display:none"></span><span
lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I have to add a Crontab
to restart the zuul and zuul-merge service and kill all
these stucked progesses or do this manually.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"
style="margin-left:0cm;text-align:left;text-indent:-18.0pt;mso-list:l0
level1 lfo1" align="left">
<!--[if !supportLists]--><span
style="font-size:10.0pt;font-family:Symbol" lang="EN-US"><span
style="mso-list:Ignore">·<span style="font:7.0pt
"Times New Roman"">
</span></span></span><!--[endif]--><span lang="EN-US">Is
there any better method to avoid the “git remote update
origin” and “ssh_wrapper” getting stuck when the network is
poor.
<o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:left" align="left"><span
lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-align:left" align="left"><span
lang="EN-US">Thanks for any input!</span></p>
</div>
</blockquote>
<br>
Hello,<br>
<br>
The zuul-merger ensures the repository is up-to-date before
attempting to merge the proposed patch. It indeeds uses 'git remote
update'. git sends a list of objects to the server which let the
server send you only the missing objects.<br>
<br>
Since the zuul-merger repositories have a reference created for each
patch merged, they are all sent to Gerrit which takes quite a while.
On Wikimedia setup it took up to 20 seconds for one of our biggest
repos see <a href="https://phabricator.wikimedia.org/T70481">https://phabricator.wikimedia.org/T70481</a>
for details.<br>
<br>
I proposed a tiny python script which let one clear out old
references from the zuul-merger repositories. It looks at zuul refs
and delete them when the commit date is older than a given amount of
days.<br>
<br>
<a href="https://review.openstack.org/#/c/109276/">https://review.openstack.org/#/c/109276/</a><br>
<br>
Safe example usage:<br>
<pre class="GKSE20JDM3 changeScreenDescription"> zuul_clear_refs.py --verbose --dry-run --until 90 /srv/zuul/git/project</pre>
<br>
Note that "git remote update" is really "git fetch --all". It might
be possible to limit the ref between fetched, ie ignore the zuul
ones when processing.<br>
<br>
You probably want to have a cronjob which repack all the ref in a
single pack ie:<br>
<br>
find /var/lib/zuul/git/ -maxdepth 3 -type d -name ".git" -exec git
--git-dir="{}" pack-refs --all \;<br>
<br>
Taken from Openstack puppet manifest for the zuul mergers:<br>
<a class="moz-txt-link-freetext" href="https://github.com/openstack-infra/puppet-zuul/blob/0f585bef4f7822975/manifests/merger.pp#L27-L35">https://github.com/openstack-infra/puppet-zuul/blob/0f585bef4f7822975/manifests/merger.pp#L27-L35</a><br>
<br>
<br>
Antoine Musso<br>
<br>
<br>
</body>
</html>