Open Stack

Sorry for the long delay in getting back to you on this, Dan. Comments inline...

On Tue, Dec 13, 2011 at 10:26 AM, Dan Prince <dan.prince at rackspace.com> wrote:
> Hi Jay,
>
> The tests should not be running concurrently.

OK, then the issue must be the fsync() timing on the servers...

> We currently have 4 Natty Cloud Servers configured as unit test workers. The
> machines are shared between nova/glance/keystone.

ok.

> The unit test runner is pretty simple:
>
> https://github.com/dprince/smokestack/blob/master/app/templates/unittest_runner.sh.erb

Yep, nothing unusual there... though I would check with Jim Blair
about the use of run_tests.sh -V -f in there. I believe Jim made some
improvements to the virtualenv setup that eliminates the need to
refresh the virtualenv (the -f flag) every time it is run...

> Would using a unique FAKE_FILESYSTEM_ROOTDIR for each test run or test help
> out here? Perhaps prefixing each one w/ /tmp/glance-tests so that we could
> also add an explicit tmp dir cleanup command in the test suite runners as
> well.

Yep, using a unique tmp dir for that would work (that's what the
functional tests in Glance do). I've added a bug report to do this:

https://bugs.launchpad.net/glance/+bug/913608

> In the meantime I'm happy to add an explicit cleanup for /tmp/glance-tests
> to make sure we have a clean slate for each test run.

I think the solution to the above bug should handle these test
failures. I should have a patch proposed shortly.

Cheers!
-jay

>
>
> Dan
>
>
>
> -----Original Message-----
> From: "Jay Pipes" <jaypipes at gmail.com>
> Sent: Tuesday, December 13, 2011 7:14am
> To: "Dan Prince" <dan.prince at rackspace.com>
> Cc: "Openstack" <openstack at lists.launchpad.net>
> Subject: Re: [Openstack] unit and integration tests results for Gerrit
>
>
> Thanks Dan, looks like a great start on this. Not sure what's going on
> with the unit test runs in Glance, though... As an example, see:
>
> http://smokestack.openstack.org/?go=/jobs/5600
>
> There are dozens of errors like this:
>
> ======================================================================
> ERROR: test_add_member (glance.tests.unit.test_api.TestGlanceAPI)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/tmp/tmp.ZaWdEoPfhP/glance_source/glance/tests/unit/test_api.py",
> line 1937, in setUp
> stubs.stub_out_filesystem_backend()
> File "/tmp/tmp.ZaWdEoPfhP/glance_source/glance/tests/stubs.py", line
> 63, in stub_out_filesystem_backend
> os.mkdir(FAKE_FILESYSTEM_ROOTDIR)
> OSError: [Errno 17] File exists: '/tmp/glance-tests'
>
> That FAKE_FILESYSTEM_ROOTDIR is cleaned up in the tearDown() method of
> API unit tests, and these tests run perfectly fine for me locally. I
> was thinking that one of the following things may be occurring:
>
> * The unlink() of the FAKE_FILESYSTEM_ROOTDIR is not fsync'ing fast
> enough, resulting in setUp()'s call to create the /tmp/glance-tests
> directory is stumbling over itself
> * The tests are being run in parallel somehow?
>
> I've seen the fsync behaviour cause havoc in some of the image cache
> tests before, and the solution ended up putting a small wait loop in
> the test code to wait until disk buffers were flushed and a cache file
> was fully removed from the filesystem. That may be happening here?
> These are being run on Cloud Servers, right?
>
> /me just trying to figure out why tests would run differently in
> smokestack than Jenkins or locally...
>
> Thanks in advance for any insight.
>
> Cheers,
> -jay