Open Stack

On 2013?01?07? 14:14, Daryl Walleck wrote:
Hi All,
   Basically, I agree with Jay. But even if we run test case using Nose 
in one process,
the setUpClass and tearDownClass methods will also run once for every 
test in the Class.
So when we run it in multiple processes, we won't increase the testing 
time. Anyway that's not
the point.
     When I resolve the parallel problem in the file 
test_server_basic_ops.py, I find all test
in this file must run in fixed order.I think there are many other tests 
like this. If "python -m
subunit.run" can support  _multiprocess_can_split_ and 
_multiprocess_can_share_ properties like
Nose, these problems will be resolved easily. If it can't, I think maybe 
we need make them one test,
not a set of tests, like Robert said. But I think it also doesn't make 
sense, because one test should
  map one interface of other projects. It will be the best, if we can 
have our code tell testr which work
can be divided, which should be treated as atomic.

Best regards
Ivan
> Hi all,
>
> Just getting caught up as I've been out of office and contact for the last few weeks.
>
> "This would indeed solve the problem of tenant/user deletion race
> conditions. BUT, and this is a very big BUT... the problem with this is
> that testr should NOT be creating X instances of some test case, one for
> each process it parallelizes to. The reason is because for the vast
> majority of Tempest test cases, the setUpClass and tearDownClass methods
> are extremely expensive and often take 90% or more of the total amount
> of time to run the test.
> ...
>
> Robert, I'm looking for some guidance from you on how to get testr to
> view the test case as the atomic unit of work for Tempest."
>
> This is the problem Nose was trying to solve with the _multiprocess_can_split_  and _multiprocess_can_share_ properties, which let the runner handle how the fixtures run. It seems like a similar solution would work here if wanted.
>
> Daryl
> ________________________________________
> From: Jay Pipes [jaypipes at gmail.com]
> Sent: Sunday, January 06, 2013 8:41 PM
> To: All Things QA.; robertc at robertcollins.net; bozhu at linux.vnet.ibm.com
> Subject: [openstack-qa] Speeding up Tempest -- testr run --parallel questions
>
> Hi Robert, Ivan, QAers,
>
> Ivan Zhu put together what seems like a trivial patch that he explains
> will get around a potential issue when running Tempest via testr run
> --parallel:
>
> https://review.openstack.org/#/c/19033/
>
> Currently, when running with allow_tenant_isolation=True (the default),
> every test case that inherits from
> tempest.tests.compute.base.BaseComputeTest gets an isolated tenant and
> user called <ClassName>-tenant and <ClassName>-user.
>
> So, for example, the ServersTestJSON test in
> tempest/tests/compute/servers/test_create_server.py will get a tenant
> called ServersTestJSON-tenant and a user called ServersTestJSON-user. At
> the end of the test, this tenant and user will be destroyed in the
> tearDownClass method.
>
> The patch from Ivan is attempting to solve a problem that running
> Tempest with testr run --parallel will spread some tests across multiple
> processes, which will result in the setUpClass and tearDownClass methods
> being run once for each process. The problem occurs when process 1
> executes the tearDownClass method and deletes the tenant for that test
> case class, process 2 might be running a test still depending on that
> tenant/user being around, which means you will run into errors in
> process 2 like this:
>
> "AuthenticationFailure: Authentication with user
> ServersTestJSON-user and password pass failed"
>
> Ivan's patch ostensibly fixes this issue by randomizing the tenant and
> user names, ensuring that tenants created for the same test in different
> processes won't be the same.
>
> This would indeed solve the problem of tenant/user deletion race
> conditions. BUT, and this is a very big BUT... the problem with this is
> that testr should NOT be creating X instances of some test case, one for
> each process it parallelizes to. The reason is because for the vast
> majority of Tempest test cases, the setUpClass and tearDownClass methods
> are extremely expensive and often take 90% or more of the total amount
> of time to run the test.
>
> So, taking ListServerFiltersTestJSON as an example, the test's
> setUpClass method creates 3 server instances. This can often take 45
> seconds or more to complete, whereas the test methods themselves (the
> dozen or so of them) complete in under 1 second total since all they are
> doing are simple list server operations. If testr "parallelizes" this
> particular test case by spreading it to 4 processes, all that has been
> done is increasing the total amount of work by a factor of 4 with no
> gain in decreased runtime, since the setUpClass method is what takes so
> darn long to complete.
>
> The real solution to this parallelization problem must be to notify
> testr that the cost of a test case is in the setUpClass call, so that it
> does not attempt to instantiate a single test case class in multiple
> processes.
>
> In other words, it should divide the total work among the processes by
> thinking of the test case class as the indivisible unit of work not the
> test method of a test case class.
>
> Robert, I'm looking for some guidance from you on how to get testr to
> view the test case as the atomic unit of work for Tempest.
>
> Thanks in advance, and sorry for the long email!
> -jay
>
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-qa/attachments/20130107/9ae02f39/attachment.html>