Open Stack

On 2015-02-25 17:02:34 +0530 (+0530), Deepak Shetty wrote:
[...]
> Run 2) We removed glusterfs backend, so Cinder was configured with
> the default storage backend i.e. LVM. We re-created the OOM here
> too
> 
> So that proves that glusterfs doesn't cause it, as its happening
> without glusterfs too.

Well, if you re-ran the job on the same VM then the second result is
potentially contaminated. Luckily this hypothesis can be confirmed
by running the second test on a fresh VM in Rackspace.

> The VM (104.239.136.99) is now in such a bad shape that existing
> ssh sessions are no longer responding for a long long time now,
> tho' ping works. So need someone to help reboot/restart the VM so
> that we can collect the logs for records. Couldn't find anyone
> during apac TZ to get it reboot.
[...]

According to novaclient that instance was in a "shutoff" state, and
so I had to nova reboot --hard to get it running. Looks like it's
back up and reachable again now.

> So from the above we can conclude that the tests are running fine
> on hpcloud and not on rax provider. Since the OS (centos7) inside
> the VM across provider is same, this now boils down to some issue
> with rax provider VM + centos7 combination.

This certainly seems possible.

> Another data point I could gather is:
>     The only other centos7 job we have is
> check-tempest-dsvm-centos7 and it does not run full tempest
> looking at the job's config it only runs smoke tests (also
> confirmed the same with Ian W) which i believe is a subset of
> tests only.

Correct, so if we confirm that we can't successfully run tempest
full on CentOS 7 in both of our providers yet, we should probably
think hard about the implications on yesterday's discussion as to
whether to set the smoke version gating on devstack and
devstack-gate changes.

> So that brings to the conclusion that probably cinder-glusterfs CI
> job (check-tempest-dsvm-full-glusterfs-centos7) is the first
> centos7 based job running full tempest tests in upstream CI and
> hence is the first to hit the issue, but on rax provider only

Entirely likely. As I mentioned last week, we don't yet have any
voting/gating jobs running on the platform as far as I can tell, so
it's still very much in an experimental stage.
-- 
Jeremy Stanley