Open Stack

Hi all,

Just getting caught up as I've been out of office and contact for the last few weeks.

"This would indeed solve the problem of tenant/user deletion race
conditions. BUT, and this is a very big BUT... the problem with this is
that testr should NOT be creating X instances of some test case, one for
each process it parallelizes to. The reason is because for the vast
majority of Tempest test cases, the setUpClass and tearDownClass methods
are extremely expensive and often take 90% or more of the total amount
of time to run the test.
...

Robert, I'm looking for some guidance from you on how to get testr to
view the test case as the atomic unit of work for Tempest."

This is the problem Nose was trying to solve with the _multiprocess_can_split_  and _multiprocess_can_share_ properties, which let the runner handle how the fixtures run. It seems like a similar solution would work here if wanted. 

Daryl
________________________________________
From: Jay Pipes [jaypipes at gmail.com]
Sent: Sunday, January 06, 2013 8:41 PM
To: All Things QA.; robertc at robertcollins.net; bozhu at linux.vnet.ibm.com
Subject: [openstack-qa] Speeding up Tempest -- testr run --parallel questions

Hi Robert, Ivan, QAers,

Ivan Zhu put together what seems like a trivial patch that he explains
will get around a potential issue when running Tempest via testr run
--parallel:

https://review.openstack.org/#/c/19033/

Currently, when running with allow_tenant_isolation=True (the default),
every test case that inherits from
tempest.tests.compute.base.BaseComputeTest gets an isolated tenant and
user called <ClassName>-tenant and <ClassName>-user.

So, for example, the ServersTestJSON test in
tempest/tests/compute/servers/test_create_server.py will get a tenant
called ServersTestJSON-tenant and a user called ServersTestJSON-user. At
the end of the test, this tenant and user will be destroyed in the
tearDownClass method.

The patch from Ivan is attempting to solve a problem that running
Tempest with testr run --parallel will spread some tests across multiple
processes, which will result in the setUpClass and tearDownClass methods
being run once for each process. The problem occurs when process 1
executes the tearDownClass method and deletes the tenant for that test
case class, process 2 might be running a test still depending on that
tenant/user being around, which means you will run into errors in
process 2 like this:

"AuthenticationFailure: Authentication with user
ServersTestJSON-user and password pass failed"

Ivan's patch ostensibly fixes this issue by randomizing the tenant and
user names, ensuring that tenants created for the same test in different
processes won't be the same.

This would indeed solve the problem of tenant/user deletion race
conditions. BUT, and this is a very big BUT... the problem with this is
that testr should NOT be creating X instances of some test case, one for
each process it parallelizes to. The reason is because for the vast
majority of Tempest test cases, the setUpClass and tearDownClass methods
are extremely expensive and often take 90% or more of the total amount
of time to run the test.

So, taking ListServerFiltersTestJSON as an example, the test's
setUpClass method creates 3 server instances. This can often take 45
seconds or more to complete, whereas the test methods themselves (the
dozen or so of them) complete in under 1 second total since all they are
doing are simple list server operations. If testr "parallelizes" this
particular test case by spreading it to 4 processes, all that has been
done is increasing the total amount of work by a factor of 4 with no
gain in decreased runtime, since the setUpClass method is what takes so
darn long to complete.

The real solution to this parallelization problem must be to notify
testr that the cost of a test case is in the setUpClass call, so that it
does not attempt to instantiate a single test case class in multiple
processes.

In other words, it should divide the total work among the processes by
thinking of the test case class as the indivisible unit of work not the
test method of a test case class.

Robert, I'm looking for some guidance from you on how to get testr to
view the test case as the atomic unit of work for Tempest.

Thanks in advance, and sorry for the long email!
-jay

_______________________________________________
openstack-qa mailing list
openstack-qa at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa