<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Ran the job manually on rax VM, provided by Jeremy. (Thank you
Jeremy).<br>
<br>
After running 971 test cases VM inaccessible for 569 ticks, then
continues... (Look at the console.log [1])<br>
And also have a look at dstat log. [2]<br>
<br>
The summary is:
<meta http-equiv="content-type" content="text/html;
charset=windows-1252">
<div id="magicdomid2372" class="ace-line" style="margin: 0px;
padding: 0px; color: rgb(0, 0, 0); font-family: monospace;
font-size: 12px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height: 16px;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span
class="author-a-iz67zz75zz76zt3nngz76zcz68zmsz90zz65z"
style="margin: 0px; padding: 1px 0px; cursor: auto;
background-color: rgb(203, 228, 85);">======</span></div>
<div id="magicdomid2373" class="ace-line" style="margin: 0px;
padding: 0px; color: rgb(0, 0, 0); font-family: monospace;
font-size: 12px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height: 16px;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span
class="author-a-iz67zz75zz76zt3nngz76zcz68zmsz90zz65z"
style="margin: 0px; padding: 1px 0px; cursor: auto;
background-color: rgb(203, 228, 85);">Totals</span></div>
<div id="magicdomid2374" class="ace-line" style="margin: 0px;
padding: 0px; color: rgb(0, 0, 0); font-family: monospace;
font-size: 12px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height: 16px;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span
class="author-a-iz67zz75zz76zt3nngz76zcz68zmsz90zz65z"
style="margin: 0px; padding: 1px 0px; cursor: auto;
background-color: rgb(203, 228, 85);">======</span></div>
<div id="magicdomid2375" class="ace-line" style="margin: 0px;
padding: 0px; color: rgb(0, 0, 0); font-family: monospace;
font-size: 12px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height: 16px;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span
class="author-a-iz67zz75zz76zt3nngz76zcz68zmsz90zz65z"
style="margin: 0px; padding: 1px 0px; cursor: auto;
background-color: rgb(203, 228, 85);">Ran: 1125 tests in
5835.0000 sec.</span></div>
<div id="magicdomid2376" class="ace-line" style="margin: 0px;
padding: 0px; color: rgb(0, 0, 0); font-family: monospace;
font-size: 12px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height: 16px;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span
class="author-a-iz67zz75zz76zt3nngz76zcz68zmsz90zz65z"
style="margin: 0px; padding: 1px 0px; cursor: auto;
background-color: rgb(203, 228, 85);"> - Passed: 960</span></div>
<div id="magicdomid2377" class="ace-line" style="margin: 0px;
padding: 0px; color: rgb(0, 0, 0); font-family: monospace;
font-size: 12px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height: 16px;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span
class="author-a-iz67zz75zz76zt3nngz76zcz68zmsz90zz65z"
style="margin: 0px; padding: 1px 0px; cursor: auto;
background-color: rgb(203, 228, 85);"> - Skipped: 88</span></div>
<div id="magicdomid2378" class="ace-line" style="margin: 0px;
padding: 0px; color: rgb(0, 0, 0); font-family: monospace;
font-size: 12px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height: 16px;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span
class="author-a-iz67zz75zz76zt3nngz76zcz68zmsz90zz65z"
style="margin: 0px; padding: 1px 0px; cursor: auto;
background-color: rgb(203, 228, 85);"> - Expected Fail: 0</span></div>
<div id="magicdomid2379" class="ace-line" style="margin: 0px;
padding: 0px; color: rgb(0, 0, 0); font-family: monospace;
font-size: 12px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height: 16px;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span
class="author-a-iz67zz75zz76zt3nngz76zcz68zmsz90zz65z"
style="margin: 0px; padding: 1px 0px; cursor: auto;
background-color: rgb(203, 228, 85);"> - Unexpected Success: 0</span></div>
<div id="magicdomid2380" class="ace-line" style="margin: 0px;
padding: 0px; color: rgb(0, 0, 0); font-family: monospace;
font-size: 12px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height: 16px;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span
class="author-a-iz67zz75zz76zt3nngz76zcz68zmsz90zz65z"
style="margin: 0px; padding: 1px 0px; cursor: auto;
background-color: rgb(203, 228, 85);"> - Failed: 77</span></div>
<div id="magicdomid2381" class="ace-line" style="margin: 0px;
padding: 0px; color: rgb(0, 0, 0); font-family: monospace;
font-size: 12px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height: 16px;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span
class="author-a-iz67zz75zz76zt3nngz76zcz68zmsz90zz65z"
style="margin: 0px; padding: 1px 0px; cursor: auto;
background-color: rgb(203, 228, 85);">Sum of execute time for
each test: 13603.6755 sec.</span></div>
<br>
<br>
[1] <a class="moz-txt-link-freetext" href="https://etherpad.openstack.org/p/rax_console.txt">https://etherpad.openstack.org/p/rax_console.txt</a><br>
[2] <a class="moz-txt-link-freetext" href="https://etherpad.openstack.org/p/rax_dstat.log">https://etherpad.openstack.org/p/rax_dstat.log</a><br>
<br>
<div class="moz-cite-prefix">On 02/24/2015 07:03 PM, Deepak Shetty
wrote:<br>
</div>
<blockquote
cite="mid:CAOXiiM=e1Ct_gS-NSvgvd4W_P2m5pLOc9oRt7OzwEe5-R1o1EQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>FWIW, we tried to run our job in a rax
provider VM (provided by ianw from his personal
account)<br>
</div>
and we ran the tempest tests twice, but the OOM
did not re-create. Of the 2 runs, one of the run<br>
</div>
used the same PYTHONHASHSEED as we had in one of the
failed runs, still no oom.<br>
<br>
</div>
Jeremy graciously agreed to provide us 2 VMs , one
each from rax and hpcloud provider<br>
</div>
to see if provider platform has anything to do with it.<br>
<br>
</div>
So we plan to run again wtih the VMs given from Jeremy ,
post which i will send<br>
</div>
next update here.<br>
<br>
</div>
thanx,<br>
</div>
deepak<br>
<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Feb 24, 2015 at 4:50 AM, Jeremy
Stanley <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:fungi@yuggoth.org" target="_blank">fungi@yuggoth.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Due to an
image setup bug (I have a fix proposed currently), I was<br>
able to rerun this on a VM in HPCloud with 30GB memory and
it<br>
completed in about an hour with a couple of tempest tests
failing.<br>
Logs at: <a moz-do-not-send="true"
href="http://fungi.yuggoth.org/tmp/logs3.tar"
target="_blank">http://fungi.yuggoth.org/tmp/logs3.tar</a><br>
<br>
Rerunning again on another 8GB Rackspace VM with the job
timeout<br>
increased to 5 hours, I was able to recreate the network<br>
connectivity issues exhibited previously. The job itself
seems to<br>
have run for roughly 3 hours while failing 15 tests, and the
worker<br>
was mostly unreachable for a while at the end (I don't know
exactly<br>
how long) until around the time it completed. The OOM
condition is<br>
present this time too according to the logs, occurring right
near<br>
the end of the job. Collected logs are available at:<br>
<a moz-do-not-send="true"
href="http://fungi.yuggoth.org/tmp/logs4.tar"
target="_blank">http://fungi.yuggoth.org/tmp/logs4.tar</a><br>
<br>
Given the comparison between these two runs, I suspect this
is<br>
either caused by memory constraints or block device I/O
performance<br>
differences (or perhaps an unhappy combination of the two).<br>
Hopefully a close review of the logs will indicate which.<br>
<div class="HOEnZb">
<div class="h5">--<br>
Jeremy Stanley<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage
questions)<br>
Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Warm Regards,
Bharat Kumar Kobagana
Software Engineer
OpenStack Storage – RedHat India
Mobile - +91 9949278005</pre>
</body>
</html>