Open Stack

Here's the benchmarks I came up with based on comparing test_server_actions with the exact same tests broken down into individual classes (see https://github.com/stackforge/cloudroast/tree/master/test_repo/compute/functional/servers/actions).

Devstack
-------------------------------------------
Current Tempest: 262.030s
Discussed Alternative (broken down into test classes and run in parallel): 98.342s

To me this design makes sense as it starts all expensive actions in parallel. I do have to use more resources to create individual servers, but in terms of runtime the effect is negligible. The fact that all the actions can also begin in parallel is where the time savings really kick in.

There's also a whole other discussion of whether test scenarios are best represented as test classes or methods, but that's more a test philosophy discussion. I'd be more than glad to discuss my views there, but it sounds like that decision has already been made.

Daryl
________________________________________
From: Giulio Fidente [gfidente at redhat.com]
Sent: Sunday, June 16, 2013 10:04 AM
To: All Things QA.
Subject: Re: [openstack-qa] Priolity brainstorming my initial toughts

On 06/16/2013 10:45 AM, Daryl Walleck wrote:
> One good example is test_server_actions under the compute tests. Optimization by reusing a single server for this class helps, but does not resolve the problem that each scenario must be run in serial. To achieve an optimal run time, each scenario should be run in parallel. What I've done in my personal testing is to break each of these down into individual test classes. While this does require additional server resources, the speedup is much higher. I'll put together some benchmarks in the morning.

this seems to me to go somehow in contrast with the problem Matthew is
trying to solve

in the ideal scenario you mention to get the best results you'll want
the setUpClass steps (which I'm currently a fan of and writing a few)
moved into the test itself so there won't be any need for it at all

I understand the logic behind it but to me it seems that it could only
produce some benefits on systems capable of very large parallelization
on both the client side AND the server side (which I don't see as a real
near-term case for us)

Having to serialize only the tests which share a number of setUpClass
steps instead seems to me a reasonable compromise

Did you manage to get any performance measurement around the two options?
--
Giulio Fidente
GPG KEY: 08D733BA | IRC: giulivo

_______________________________________________
openstack-qa mailing list
openstack-qa at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa