Open Stack

On Mon, Jun 17, 2013 at 10:06:04AM +0200, Giulio Fidente wrote:
> On 06/17/2013 12:42 AM, Daryl Walleck wrote:
> >Here's the benchmarks I came up with based on comparing test_server_actions with the exact same tests broken down into individual classes (see https://github.com/stackforge/cloudroast/tree/master/test_repo/compute/functional/servers/actions).
> >
> >Devstack
> >-------------------------------------------
> >Current Tempest: 262.030s
> >Discussed Alternative (broken down into test classes and run in parallel): 98.342s
> 
> Daryl, that's ~3 times faster but are you parallelizing the
> execution of the classes or of the test cases (duplicating the
> setUpClass steps)?
> 
> In his email Matthew suggests that, given our current usage of
> setUpClass, parallelizing every single test case won't produce
> better results.

So what Daryl is describing here is a slightly different type of resource issue
then what I've been looking at. I was talking about tests like the lists tests
where we have multiple time consuming creates (and deletes with waits) for
relatively fast tests. This is basically the converse case where we
have multiple expensive actions in the test and a single shared resource (which
is relatively fast to create) between them. I think this is probably the
minority of what we have going on in tempest. (although certain actions are the
perfect example) Thinking about it though this is essentially what should be
happening right now with 'testr run --parallel' for these types of test cases.

Right now IMO we should probably be looking at getting the class level
partitioning first and once we get that moving we can start optimizing things
further. I'm trying to avoid making large test restructures up front as a first
step for speeding things up. (It's probably unavoidable to a certain degree
though)

> 
> To leverage the classes parallelization instead, after the changes
> in testr will be available, maybe we could just revisit the tests
> and try to split them across more classes to allow for better
> partitioning.

I agree, there will definitely be further optimizations down the road. If this
ends up being a large bottleneck it's definitely worth investigating and
restructuring our tests.

-Matt Treinish