<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 2013年01月07日 14:14, Daryl Walleck
wrote:<br>
Hi All,<br>
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
Basically, I agree with Jay. But even if we run test case using
Nose in one process,<br>
the setUpClass and tearDownClass methods will also run once for
every test in the Class.<br>
So when we run it in multiple processes, we won't increase the
testing time. Anyway that's not<br>
the point.<br>
When I resolve the parallel problem in the file
test_server_basic_ops.py, I find all test<br>
in this file must run in fixed order.<span style="color: rgb(0, 0,
0); font-family: Arial; font-size: 14px; font-style: normal;
font-variant: normal; font-weight: normal; letter-spacing:
normal; line-height: 22px; orphans: 2; text-align: -webkit-auto;
text-indent: 0px; text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); display: inline !important; float: none; "></span> I think
there are many other tests like this. If "python -m<br>
subunit.run" can support _multiprocess_can_split_ and
_multiprocess_can_share_ properties like<br>
Nose, these problems will be resolved easily. If it can't, I think
maybe we need make them one test,<br>
not a set of tests, like Robert said. But I think it also doesn't
make sense, because one test should<br>
map one interface of other projects. It will be the best, if we
can have our code tell testr which work<br>
can be divided, which should be treated as
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
atomic.<br>
<br>
Best regards <br>
Ivan<br>
</div>
<blockquote
cite="mid:BF1FE6A2EBA2D54BBF44C3128B132D9C65E75ABF@ORD1EXD04.RACKSPACE.CORP"
type="cite">
<pre wrap="">Hi all,
Just getting caught up as I've been out of office and contact for the last few weeks.
"This would indeed solve the problem of tenant/user deletion race
conditions. BUT, and this is a very big BUT... the problem with this is
that testr should NOT be creating X instances of some test case, one for
each process it parallelizes to. The reason is because for the vast
majority of Tempest test cases, the setUpClass and tearDownClass methods
are extremely expensive and often take 90% or more of the total amount
of time to run the test.
...
Robert, I'm looking for some guidance from you on how to get testr to
view the test case as the atomic unit of work for Tempest."
This is the problem Nose was trying to solve with the _multiprocess_can_split_ and _multiprocess_can_share_ properties, which let the runner handle how the fixtures run. It seems like a similar solution would work here if wanted.
Daryl
________________________________________
From: Jay Pipes [<a class="moz-txt-link-abbreviated" href="mailto:jaypipes@gmail.com">jaypipes@gmail.com</a>]
Sent: Sunday, January 06, 2013 8:41 PM
To: All Things QA.; <a class="moz-txt-link-abbreviated" href="mailto:robertc@robertcollins.net">robertc@robertcollins.net</a>; <a class="moz-txt-link-abbreviated" href="mailto:bozhu@linux.vnet.ibm.com">bozhu@linux.vnet.ibm.com</a>
Subject: [openstack-qa] Speeding up Tempest -- testr run --parallel questions
Hi Robert, Ivan, QAers,
Ivan Zhu put together what seems like a trivial patch that he explains
will get around a potential issue when running Tempest via testr run
--parallel:
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/19033/">https://review.openstack.org/#/c/19033/</a>
Currently, when running with allow_tenant_isolation=True (the default),
every test case that inherits from
tempest.tests.compute.base.BaseComputeTest gets an isolated tenant and
user called <ClassName>-tenant and <ClassName>-user.
So, for example, the ServersTestJSON test in
tempest/tests/compute/servers/test_create_server.py will get a tenant
called ServersTestJSON-tenant and a user called ServersTestJSON-user. At
the end of the test, this tenant and user will be destroyed in the
tearDownClass method.
The patch from Ivan is attempting to solve a problem that running
Tempest with testr run --parallel will spread some tests across multiple
processes, which will result in the setUpClass and tearDownClass methods
being run once for each process. The problem occurs when process 1
executes the tearDownClass method and deletes the tenant for that test
case class, process 2 might be running a test still depending on that
tenant/user being around, which means you will run into errors in
process 2 like this:
"AuthenticationFailure: Authentication with user
ServersTestJSON-user and password pass failed"
Ivan's patch ostensibly fixes this issue by randomizing the tenant and
user names, ensuring that tenants created for the same test in different
processes won't be the same.
This would indeed solve the problem of tenant/user deletion race
conditions. BUT, and this is a very big BUT... the problem with this is
that testr should NOT be creating X instances of some test case, one for
each process it parallelizes to. The reason is because for the vast
majority of Tempest test cases, the setUpClass and tearDownClass methods
are extremely expensive and often take 90% or more of the total amount
of time to run the test.
So, taking ListServerFiltersTestJSON as an example, the test's
setUpClass method creates 3 server instances. This can often take 45
seconds or more to complete, whereas the test methods themselves (the
dozen or so of them) complete in under 1 second total since all they are
doing are simple list server operations. If testr "parallelizes" this
particular test case by spreading it to 4 processes, all that has been
done is increasing the total amount of work by a factor of 4 with no
gain in decreased runtime, since the setUpClass method is what takes so
darn long to complete.
The real solution to this parallelization problem must be to notify
testr that the cost of a test case is in the setUpClass call, so that it
does not attempt to instantiate a single test case class in multiple
processes.
In other words, it should divide the total work among the processes by
thinking of the test case class as the indivisible unit of work not the
test method of a test case class.
Robert, I'm looking for some guidance from you on how to get testr to
view the test case as the atomic unit of work for Tempest.
Thanks in advance, and sorry for the long email!
-jay
_______________________________________________
openstack-qa mailing list
<a class="moz-txt-link-abbreviated" href="mailto:openstack-qa@lists.openstack.org">openstack-qa@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a>
</pre>
</blockquote>
<br>
</body>
</html>