<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=gb2312">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:SimSun;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:SimSun;}
tt
{mso-style-priority:99;
font-family:SimSun;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:SimSun;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:457604237;
mso-list-type:hybrid;
mso-list-template-ids:1983965060 3178246 269025283 269025285 269025281 269025283 269025285 269025281 269025283 269025285;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;
mso-fareast-font-family:SimSun;
mso-bidi-font-family:"Times New Roman";}
@list l1
{mso-list-id:1365131896;
mso-list-type:hybrid;
mso-list-template-ids:2137540060 -2056457042 269025283 269025285 269025281 269025283 269025285 269025281 269025283 269025285;}
@list l1:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;
mso-fareast-font-family:SimSun;
mso-bidi-font-family:"Times New Roman";}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-CA" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">As Kai Qiang mentioned, magnum gate recently had a bunch of random failures, which occurred on creating a nova instance with 2G of RAM. According to the error
message, it seems that the hypervisor tried to allocate memory to the nova instance but couldn’t find enough free memory in the host. However, by adding a few “nova hypervisor-show XX” before, during, and right after the test, it showed that the host has 6G
of free RAM, which is far more than 2G. Here is a snapshot of the output [1]. You can find the full log here [2].<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Another observation is that most of the failure happened on a node with name “devstack-trusty-ovh-*” (You can verify it by entering a query [3] at
<a href="http://logstash.openstack.org/">http://logstash.openstack.org/</a> ). It seems that the jobs will be fine if they are allocated to a node other than “ovh”.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Any hints to debug this issue further? Suggestions are greatly appreciated.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">[1]
<a href="http://paste.openstack.org/show/481746/">http://paste.openstack.org/show/481746/</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">[2]
<a href="http://logs.openstack.org/48/256748/1/check/gate-functional-dsvm-magnum-swarm/56d79c3/console.html">
http://logs.openstack.org/48/256748/1/check/gate-functional-dsvm-magnum-swarm/56d79c3/console.html</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">[3]
<a href="https://review.openstack.org/#/c/254370/2/queries/1521237.yaml">https://review.openstack.org/#/c/254370/2/queries/1521237.yaml</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Best regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hongbin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Kai Qiang Wu [mailto:wkqwu@cn.ibm.com]
<br>
<b>Sent:</b> December-09-15 7:23 AM<br>
<b>To:</b> openstack-dev@lists.openstack.org<br>
<b>Subject:</b> [openstack-dev] [Infra][nova][magnum] Jenkins failed quite often for "Cannot set up guest memory 'pc.ram': Cannot allocate memory"<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>Hi All,<br>
<br>
I am not sure what changes these days, We found quite often now, the Jenkins failed for:<br>
<br>
<br>
<tt><span style="font-size:13.5pt"><a href="http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz">http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz</a></span></tt><br>
<br>
<a href="http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_892"><tt><span style="font-size:13.5pt;color:#A11F12">2015-12-09 08:52:27.892</span></tt></a><tt><span style="font-size:13.5pt">+0000:
22957: debug : qemuMonitorJSONCommandWithFd:264 : Send command '{"execute":"qmp_capabilities","id":"libvirt-1"}' for write with FD -1</span></tt><br>
<a href="http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_892"><tt><span style="font-size:13.5pt;color:#A11F12">2015-12-09 08:52:27.892</span></tt></a><tt><span style="font-size:13.5pt">+0000:
22957: debug : qemuMonitorSend:959 : QEMU_MONITOR_SEND_MSG: mon=0x7fa66400c6f0 msg={"execute":"qmp_capabilities","id":"libvirt-1"}</span></tt><br>
<tt><span style="font-size:13.5pt"> fd=-1</span></tt><br>
<a href="http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941"><tt><span style="font-size:13.5pt;color:#A11F12">2015-12-09 08:52:27.941</span></tt></a><tt><span style="font-size:13.5pt">+0000:
22951: debug : virNetlinkEventCallback:347 : dispatching to max 0 clients, called from event watch 6</span></tt><br>
<a href="http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941"><tt><span style="font-size:13.5pt;color:#A11F12">2015-12-09 08:52:27.941</span></tt></a><tt><span style="font-size:13.5pt">+0000:
22951: debug : virNetlinkEventCallback:360 : event not handled.</span></tt><br>
<a href="http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941"><tt><span style="font-size:13.5pt;color:#A11F12">2015-12-09 08:52:27.941</span></tt></a><tt><span style="font-size:13.5pt">+0000:
22951: debug : virNetlinkEventCallback:347 : dispatching to max 0 clients, called from event watch 6</span></tt><br>
<a href="http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941"><tt><span style="font-size:13.5pt;color:#A11F12">2015-12-09 08:52:27.941</span></tt></a><tt><span style="font-size:13.5pt">+0000:
22951: debug : virNetlinkEventCallback:360 : event not handled.</span></tt><br>
<a href="http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941"><tt><span style="font-size:13.5pt;color:#A11F12">2015-12-09 08:52:27.941</span></tt></a><tt><span style="font-size:13.5pt">+0000:
22951: debug : virNetlinkEventCallback:347 : dispatching to max 0 clients, called from event watch 6</span></tt><br>
<a href="http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941"><tt><span style="font-size:13.5pt;color:#A11F12">2015-12-09 08:52:27.941</span></tt></a><tt><span style="font-size:13.5pt">+0000:
22951: debug : virNetlinkEventCallback:360 : event not handled.</span></tt><br>
<a href="http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_28_070"><tt><span style="font-size:13.5pt;color:#A11F12">2015-12-09 08:52:28.070</span></tt></a><tt><span style="font-size:13.5pt">+0000:
22951: error : qemuMonitorIORead:554 : Unable to read from monitor: Connection reset by peer</span></tt><br>
<a href="http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_28_070"><tt><span style="font-size:13.5pt;color:#A11F12">2015-12-09 08:52:28.070</span></tt></a><tt><span style="font-size:13.5pt">+0000:
22951: error : qemuMonitorIO:690 : internal error: early end of file from monitor: possible problem:</span></tt><br>
<tt><span style="font-size:13.5pt">Cannot set up guest memory 'pc.ram': Cannot allocate memory</span></tt><br>
<br>
<br>
<br>
<br>
<span style="font-size:13.5pt;color:#0020C2">It not hit such resource issue before. I am not sure if Infra or nova guys know about it ?
</span><br>
<br>
<br>
Thanks<br>
<br>
Best Wishes,<br>
--------------------------------------------------------------------------------<br>
Kai Qiang Wu (<span lang="ZH-CN">吴开强</span> Kennan<span lang="ZH-CN">）</span><br>
IBM China System and Technology Lab, Beijing<br>
<br>
E-mail: <a href="mailto:wkqwu@cn.ibm.com">wkqwu@cn.ibm.com</a><br>
Tel: 86-10-82451647<br>
Address: Building 28(Ring Building), ZhongGuanCun Software Park, <br>
No.8 Dong Bei Wang West Road, Haidian District Beijing P.R.China 100193<br>
--------------------------------------------------------------------------------<br>
Follow your heart. You are miracle! <o:p></o:p></p>
</div>
</body>
</html>