<div dir="ltr">Hey all,<div><br></div><div>We've made some progress with the gates this past week. There are still some issues, but I want to point out that I've also seen a lot of real errors get a recheck comment recently. It slows the gate down and wastes infra quota to recheck things that are going to fail again. Can I suggest that we all make sure to get back in the habit of looking at failures and noting down a reason for the recheck? This will also help track what issues still remain to be fixed with the gates.</div><div><br></div><div>Thanks,</div><div>Corey</div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Feb 8, 2016 at 12:10 PM Hongbin Lu <<a href="mailto:hongbin.lu@huawei.com">hongbin.lu@huawei.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-CA" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hi Team,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">In order to resolve issue #3, it looks like we have to significantly reduce the memory consumption of the gate tests. Details can be found in this patch
<a href="https://review.openstack.org/#/c/276958/" target="_blank">https://review.openstack.org/#/c/276958/</a> . For core team, a fast review and approval of that patch would be greatly appreciated, since it is hard to work with a gate that takes several hours to complete.
Thanks.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Best regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hongbin<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Corey O'Brien [mailto:<a href="mailto:coreypobrien@gmail.com" target="_blank">coreypobrien@gmail.com</a>]
<br>
<b>Sent:</b> February-05-16 12:04 AM</span></p></div></div></div><div lang="EN-CA" link="blue" vlink="purple"><div><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm"><p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""><br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
</span></p></div></div></div><div lang="EN-CA" link="blue" vlink="purple"><div><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm"><p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""><b>Subject:</b> [openstack-dev] [Magnum] gate issues<u></u><u></u></span></p>
</div></div></div><div lang="EN-CA" link="blue" vlink="purple"><div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">So as we're all aware, the gate is a mess right now. I wanted to sum up some of the issues so we can figure out solutions.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">1. The functional-api job sometimes fails because bays timeout building after 1 hour. The logs look something like this:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";color:black">magnum.tests.functional.api.v1.test_bay.BayTest.test_create_list_and_delete_bays [3733.626171s] ... FAILED</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I can reproduce this hang on my devstack with etcdctl 2.0.10 as described in this bug (<a href="https://bugs.launchpad.net/magnum/+bug/1541105" target="_blank">https://bugs.launchpad.net/magnum/+bug/1541105</a>), but apparently either my fix with using
2.2.5 (<a href="https://review.openstack.org/#/c/275994/" target="_blank">https://review.openstack.org/#/c/275994/</a>) is incomplete or there is another intermittent problem because it happened again even with that fix: (<a href="http://logs.openstack.org/94/275994/1/check/gate-functional-dsvm-magnum-api/32aacb1/console.html" target="_blank">http://logs.openstack.org/94/275994/1/check/gate-functional-dsvm-magnum-api/32aacb1/console.html</a>)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">2. The k8s job has some sort of intermittent hang as well that causes a similar symptom as with swarm. <a href="https://bugs.launchpad.net/magnum/+bug/1541964" target="_blank">https://bugs.launchpad.net/magnum/+bug/1541964</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">3. When the functional-api job runs, it frequently destroys the VM causing the jenkins slave agent to die. Example:
<a href="http://logs.openstack.org/03/275003/6/check/gate-functional-dsvm-magnum-api/a9a0eb9/console.html" target="_blank">
http://logs.openstack.org/03/275003/6/check/gate-functional-dsvm-magnum-api/a9a0eb9//console.html</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">When this happens, zuul re-queues a new build from the start on a new VM. This can happen many times in a row before the job completes.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I chatted with openstack-infra about this and after taking a look at one of the VMs, it looks like memory over consumption leading to thrashing was a possible culprit. The sshd daemon was also dead but the console showed things like "INFO:
task kswapd0:77 blocked for more than 120 seconds". A cursory glance and following some of the jobs seems to indicate that this doesn't happen on RAX VMs which have swap devices unlike the OVH VMs as well.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">4. In general, even when things work, the gate is really slow. The sequential master-then-node build process in combination with underpowered VMs makes bay builds take 25-30 minutes when they do succeed. Since we're already close to tipping
over a VM, we run functional tests with concurrency=1, so 2 bay builds means almost the entire allotted devstack testing time (generally 75 minutes of actual test time available it seems).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Corey<u></u><u></u></p>
</div>
</div>
</div></div><div lang="EN-CA" link="blue" vlink="purple"><div></div>
</div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div>