<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
        {font-family:Helvetica;
        panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
        {font-family:SimSun;
        panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
        {font-family:SimSun;
        panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
        {font-family:Tahoma;
        panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
        {font-family:"\@SimSun";
        panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0cm;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:purple;
        text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
        {mso-style-priority:99;
        mso-style-link:"Balloon Text Char";
        margin:0cm;
        margin-bottom:.0001pt;
        font-size:8.0pt;
        font-family:"Tahoma","sans-serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
        {mso-style-priority:34;
        margin-top:0cm;
        margin-right:0cm;
        margin-bottom:0cm;
        margin-left:36.0pt;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman","serif";}
span.BalloonTextChar
        {mso-style-name:"Balloon Text Char";
        mso-style-priority:99;
        mso-style-link:"Balloon Text";
        font-family:"Tahoma","sans-serif";}
span.EmailStyle19
        {mso-style-type:personal-reply;
        font-family:"Calibri","sans-serif";
        color:#1F497D;}
.MsoChpDefault
        {mso-style-type:export-only;
        font-size:10.0pt;}
@page WordSection1
        {size:612.0pt 792.0pt;
        margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
        {page:WordSection1;}
/* List Definitions */
@list l0
        {mso-list-id:240602453;
        mso-list-type:hybrid;
        mso-list-template-ids:-907131520 269025295 269025305 269025307 269025295 269025305 269025307 269025295 269025305 269025307;}
@list l0:level1
        {mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-18.0pt;}
ol
        {margin-bottom:0cm;}
ul
        {margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-CA" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Corey,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks for investigating the gate issues and summarizing it. It looks there are multiple problems to solve, and tickets were created for each one.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">      
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><a href="https://bugs.launchpad.net/magnum/+bug/1542384">https://bugs.launchpad.net/magnum/+bug/1542384</a><o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">      
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><a href="https://bugs.launchpad.net/magnum/+bug/1541964">https://bugs.launchpad.net/magnum/+bug/1541964</a><o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><span style="mso-list:Ignore">3.<span style="font:7.0pt "Times New Roman"">      
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><a href="https://bugs.launchpad.net/magnum/+bug/1542386">https://bugs.launchpad.net/magnum/+bug/1542386</a><o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><span style="mso-list:Ignore">4.<span style="font:7.0pt "Times New Roman"">      
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><a href="https://bugs.launchpad.net/magnum/+bug/1536739">https://bugs.launchpad.net/magnum/+bug/1536739</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I gave #3 the highest priority because, without this issue being resolved, the gate takes several hours to run a single job. It would be tedious for testing
 patches and trouble-shooting other issues in such environment. Any kind of help for this issue is greatly appreciated.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Egor, thanks for the advice. A ticket was created to track the logs missing issue you mentioned: https://bugs.launchpad.net/magnum/+bug/1542390<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Best regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hongbin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Guz Egor [mailto:guz_egor@yahoo.com]
<br>
<b>Sent:</b> February-05-16 2:44 AM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [Magnum] gate issues<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div id="yui_3_16_0_1_1454620984159_85345">
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial","sans-serif";color:black">Corey,</span><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:black"><o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85355">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:black"> <o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85355">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:black">I think we should do more investigation before applying any "hot" patches. E.g. I look at several failures today and honestly there
 is no way to find out reasons.<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85355">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:black">I believe we are not copying logs (<a href="https://github.com/openstack/magnum/blob/master/magnum/tests/functional/python_client_base.py#L163">https://github.com/openstack/magnum/blob/master/magnum/tests/functional/python_client_base.py#L163</a>)
 during test failure,  <o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85355">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:black">we register handler at setUp (<a href="https://github.com/openstack/magnum/blob/master/magnum/tests/functional/python_client_base.py#L244">https://github.com/openstack/magnum/blob/master/magnum/tests/functional/python_client_base.py#L244</a>),
 but Swarm tests, create<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85355">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:black">bay in setUpClass (<a href="https://github.com/openstack/magnum/blob/master/magnum/tests/functional/swarm/test_swarm_python_client.py#L48">https://github.com/openstack/magnum/blob/master/magnum/tests/functional/swarm/test_swarm_python_client.py#L48</a>)
 which called before setUp.<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85355">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:black">So there is no way to see any logs from vm.<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85355">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85355">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:black">sorry, I cannot submit patch/debug by myself because I will get my laptop back only on Tue ):<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85355">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85354">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:black">--- <o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85354">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:black"> Egor<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85346">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85351">
<div id="yui_3_16_0_1_1454620984159_85350">
<div id="yui_3_16_0_1_1454620984159_85349">
<div id="yui_3_16_0_1_1454620984159_85348">
<div class="MsoNormal" align="center" style="text-align:center;background:white">
<span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">
<hr size="1" width="100%" align="center">
</span></div>
<p class="MsoNormal" style="background:white"><b><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">From:</span></b><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black"> Corey O'Brien <<a href="mailto:coreypobrien@gmail.com">coreypobrien@gmail.com</a>><br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions) <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>>
<br>
<b>Sent:</b> Thursday, February 4, 2016 9:03 PM<br>
<b>Subject:</b> [openstack-dev] [Magnum] gate issues</span><span style="font-family:"Helvetica","sans-serif";color:black"><o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85352">
<p class="MsoNormal" style="background:white"><span style="font-family:"Helvetica","sans-serif";color:black"><o:p> </o:p></span></p>
<div id="yiv3432290026">
<div id="yui_3_16_0_1_1454620984159_85421">
<p class="MsoNormal" style="background:white"><span style="font-family:"Helvetica","sans-serif";color:black">So as we're all aware, the gate is a mess right now. I wanted to sum up some of the issues so we can figure out solutions.<o:p></o:p></span></p>
<div id="yui_3_16_0_1_1454620984159_86205">
<p class="MsoNormal" style="background:white"><span style="font-family:"Helvetica","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85424">
<p class="MsoNormal" style="background:white"><span style="font-family:"Helvetica","sans-serif";color:black">1. The functional-api job sometimes fails because bays timeout building after 1 hour. The logs look something like this:<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85541">
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial","sans-serif";color:black">magnum.tests.functional.api.v1.test_bay.BayTest.test_create_list_and_delete_bays [3733.626171s] ... FAILED</span><span style="font-family:"Helvetica","sans-serif";color:black"><o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85428">
<p class="MsoNormal" style="background:white"><span style="font-family:"Helvetica","sans-serif";color:black">I can reproduce this hang on my devstack with etcdctl 2.0.10 as described in this bug (<a href="https://bugs.launchpad.net/magnum/+bug/1541105" target="_blank">https://bugs.launchpad.net/magnum/+bug/1541105</a>),
 but apparently either my fix with using 2.2.5 (<a href="https://review.openstack.org/#/c/275994/" target="_blank" id="yui_3_16_0_1_1454620984159_85546">https://review.openstack.org/#/c/275994/</a>) is incomplete or there is another intermittent problem because
 it happened again even with that fix: (<a href="http://logs.openstack.org/94/275994/1/check/gate-functional-dsvm-magnum-api/32aacb1/console.html" target="_blank" id="yui_3_16_0_1_1454620984159_85840">http://logs.openstack.org/94/275994/1/check/gate-functional-dsvm-magnum-api/32aacb1/console.html</a>)<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85611">
<p class="MsoNormal" style="background:white"><span style="font-family:"Helvetica","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85542">
<p class="MsoNormal" style="background:white"><span style="font-family:"Helvetica","sans-serif";color:black">2. The k8s job has some sort of intermittent hang as well that causes a similar symptom as with swarm. <a href="https://bugs.launchpad.net/magnum/+bug/1541964" target="_blank">https://bugs.launchpad.net/magnum/+bug/1541964</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Helvetica","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Helvetica","sans-serif";color:black">3. When the functional-api job runs, it frequently destroys the VM causing the jenkins slave agent to die. Example:
<a href="http://logs.openstack.org/03/275003/6/check/gate-functional-dsvm-magnum-api/a9a0eb9/console.html" target="_blank">
http://logs.openstack.org/03/275003/6/check/gate-functional-dsvm-magnum-api/a9a0eb9//console.html</a><o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1454620984159_85456">
<p class="MsoNormal" style="background:white"><span style="font-family:"Helvetica","sans-serif";color:black">When this happens, zuul re-queues a new build from the start on a new VM. This can happen many times in a row before the job completes.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Helvetica","sans-serif";color:black">I chatted with openstack-infra about this and after taking a look at one of the VMs, it looks like memory over consumption leading to thrashing was
 a possible culprit. The sshd daemon was also dead but the console showed things like "INFO: task kswapd0:77 blocked for more than 120 seconds". A cursory glance and following some of the jobs seems to indicate that this doesn't happen on RAX VMs which have
 swap devices unlike the OVH VMs as well.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Helvetica","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Helvetica","sans-serif";color:black">4. In general, even when things work, the gate is really slow. The sequential master-then-node build process in combination with underpowered VMs makes
 bay builds take 25-30 minutes when they do succeed. Since we're already close to tipping over a VM, we run functional tests with concurrency=1, so 2 bay builds means almost the entire allotted devstack testing time (generally 75 minutes of actual test time
 available it seems).<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Helvetica","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Helvetica","sans-serif";color:black">Corey<o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt;background:white"><span style="font-family:"Helvetica","sans-serif";color:black"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org">OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>