<div dir="ltr"><div><div><div><div><div>Hi Jeremy,<br></div> Couldn't find anything strong in the logs to back the reason for OOM.<br></div>At the time OOM happens, mysqld and java processes have the most RAM hence OOM selects mysqld (4.7G) to be killed.<br><br></div>From a glusterfs backend perspective, i haven't found anything suspicious, and we don't have the logs of glusterfs (which is typically in /var/log/glusterfs) so can't delve inside glusterfs too much :(<br><br></div><div>BharatK (in CC) also tried to re-create the issue in local VM setup, but it hasn't yet!<br></div><div><br>Having said that,<u><b> we do know</b></u> that we started seeing this issue after we enabled the nova-assisted-snapshot tests (by changing nova' s policy.json to enable non-admin to create hyp-assisted snaps). We think that enabling online snaps might have added to the number of tests and memory load & thats the only clue we have as of now!<br><br>So :<br><br> 1) BharatK has merged the patch ( <a href="https://review.openstack.org/#/c/157707/">https://review.openstack.org/#/c/157707/</a> ) to revert the policy.json in the glusterfs job. So no more nova-assisted-snap tests.<br></div><br> 2) We also are increasing the timeout of our job in patch ( <a href="https://review.openstack.org/#/c/157835/1">https://review.openstack.org/#/c/157835/1</a> ) so that we can get a full run without timeouts to do a good analysis of the logs (logs are not posted if the job times out)<br><br></div>Can you please re-enable our job, so that we can confirm that disabling online snap TCs is helping the issue, which if it does, can help us narrow down the issue.<br><br>We also plan to monitor & debug over the weekend hence having the job enabled can help us a lot.<br><br>thanx,<br>deepak<br><div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 19, 2015 at 10:37 PM, Jeremy Stanley <span dir="ltr"><<a href="mailto:fungi@yuggoth.org" target="_blank">fungi@yuggoth.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 2015-02-19 17:03:49 +0100 (+0100), Deepak Shetty wrote:<br>
[...]<br>
<span class="">> For some reason we are seeing the centos7 glusterfs CI job getting<br>
</span>> aborted/ killed either by Java exception or the build getting<br>
> aborted due to timeout.<br>
[...]<br>
<span class="">> Hoping to root cause this soon and get the cinder-glusterfs CI job<br>
> back online soon.<br>
<br>
</span>I manually reran the same commands this job runs on an identical<br>
virtual machine and was able to reproduce some substantial<br>
weirdness.<br>
<br>
I temporarily lost remote access to the VM around 108 minutes into<br>
running the job (~17:50 in the logs) and the out of band console<br>
also became unresponsive to carriage returns. The machine's IP<br>
address still responded to ICMP ping, but attempts to open new TCP<br>
sockets to the SSH service never got a protocol version banner back.<br>
After about 10 minutes of that I went out to lunch but left<br>
everything untouched. To my excitement it was up and responding<br>
again when I returned.<br>
<br>
It appears from the logs that it runs well past the 120-minute mark<br>
where devstack-gate tries to kill the gate hook for its configured<br>
timeout. Somewhere around 165 minutes in (18:47) you can see the<br>
kernel out-of-memory killer starts to kick in and kill httpd and<br>
mysqld processes according to the syslog. Hopefully this is enough<br>
additional detail to get you a start at finding the root cause so<br>
that we can reenable your job. Let me know if there's anything else<br>
you need for this.<br>
<br>
[1] <a href="http://fungi.yuggoth.org/tmp/logs.tar" target="_blank">http://fungi.yuggoth.org/tmp/logs.tar</a><br>
<span class="HOEnZb"><font color="#888888">--<br>
Jeremy Stanley<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</font></span></blockquote></div><br></div>