[openstack-dev] [devstack] [Cinder-GlusterFS CI] centos7 gate job abrupt failures

Bharat Kumar bharat.kobagana at redhat.com
Tue Feb 24 19:32:07 UTC 2015


Ran the job manually on rax VM, provided by Jeremy. (Thank you Jeremy).

After running 971 test cases VM inaccessible for 569 ticks, then 
continues... (Look at the console.log [1])
And also have a look at dstat log. [2]

The summary is:
======
Totals
======
Ran: 1125 tests in 5835.0000 sec.
  - Passed: 960
  - Skipped: 88
  - Expected Fail: 0
  - Unexpected Success: 0
  - Failed: 77
Sum of execute time for each test: 13603.6755 sec.


[1] https://etherpad.openstack.org/p/rax_console.txt
[2] https://etherpad.openstack.org/p/rax_dstat.log

On 02/24/2015 07:03 PM, Deepak Shetty wrote:
> FWIW, we tried to run our job in a rax provider VM (provided by ianw 
> from his personal account)
> and we ran the tempest tests twice, but the OOM did not re-create. Of 
> the 2 runs, one of the run
> used the same PYTHONHASHSEED as we had in one of the failed runs, 
> still no oom.
>
> Jeremy graciously agreed to provide us 2 VMs , one each from rax and 
> hpcloud provider
> to see if provider platform has anything to do with it.
>
> So we plan to run again wtih the VMs given from Jeremy , post which i 
> will send
> next update here.
>
> thanx,
> deepak
>
>
> On Tue, Feb 24, 2015 at 4:50 AM, Jeremy Stanley <fungi at yuggoth.org 
> <mailto:fungi at yuggoth.org>> wrote:
>
>     Due to an image setup bug (I have a fix proposed currently), I was
>     able to rerun this on a VM in HPCloud with 30GB memory and it
>     completed in about an hour with a couple of tempest tests failing.
>     Logs at: http://fungi.yuggoth.org/tmp/logs3.tar
>
>     Rerunning again on another 8GB Rackspace VM with the job timeout
>     increased to 5 hours, I was able to recreate the network
>     connectivity issues exhibited previously. The job itself seems to
>     have run for roughly 3 hours while failing 15 tests, and the worker
>     was mostly unreachable for a while at the end (I don't know exactly
>     how long) until around the time it completed. The OOM condition is
>     present this time too according to the logs, occurring right near
>     the end of the job. Collected logs are available at:
>     http://fungi.yuggoth.org/tmp/logs4.tar
>
>     Given the comparison between these two runs, I suspect this is
>     either caused by memory constraints or block device I/O performance
>     differences (or perhaps an unhappy combination of the two).
>     Hopefully a close review of the logs will indicate which.
>     --
>     Jeremy Stanley
>
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
Warm Regards,
Bharat Kumar Kobagana
Software Engineer
OpenStack Storage – RedHat India
Mobile - +91 9949278005

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150225/626d46b9/attachment.html>


More information about the OpenStack-dev mailing list