Open Stack

Matt,

The per class sounds good to me.

Another partitioning possibility is using testresources
 for the 'active servers',
Most of the test cases just needs an active server,
 and in the past it was the most expensive operation.

The build interval in the past was 3 sec, now it is 1 sec,
 this also can have effect to the total active server cost.
 
testresources has concepts to work in higher scope than a class.

The per class or test method isolated credentials
 can cause issues in this case.

I am wondering what is missing for partitioning active servers
 equally among the worker processes ?
What are the other expensive operations which enforcing serialization.
Is the active volumes ?

OptimisingTestSuite seams like the suite which needed to mange the
 resources over classes.
Can it be used together ?

May be this the option we did not considered yet.

We should use the similar sized system as gate jobs for benchmarking. 

I think the speed-up is one of the most important project now.
You are the leader of this project, anything I said before is just recommendation. 

I am starting to hunt down all kind of resource leakage, what I can detect.
If there is anything else what I can do let me know. 

Many of the gate jobs fails because of "race" issues in the other services.
Finding the root cause is usually not a small task,
 but this is another thing I think I can help.

Best Regards,
Attila

PS.: in the past tempest code base contained parallel unsafe,
  invalid token injection. Is this issue completely solved ?


----- Original Message -----
From: "Matthew Treinish" <mtreinish at kortar.org>
To: "All Things QA." <openstack-qa at lists.openstack.org>
Sent: Friday, June 14, 2013 10:13:24 PM
Subject: Re: [openstack-qa] Priolity brainstorming my initial toughts

On Fri, Jun 14, 2013 at 09:12:11PM +0200, Giulio Fidente wrote:
> On 06/14/2013 10:33 AM, Attila Fazekas wrote:
> >- Gate time is continuously increasing and it will
> >   if we do not act soon.
> 
> It's an interesting topic but I need a little wrap on on the
> situation. Still, it'll hopefully be useful to other people also
> willing to contribute.
> 
> What are the blockers currently preventing us from using testr and
> "drop" nose?

So testr will technically run today, (to test it use 'testr run' or in parallel
'testr run --parallel') but you'll find that it is slower than just running
serially with nose. It also won't pass all the tests but that's a different
problem to solve after this one. The problem is with setUpClass(), in tempest
for a lot of tests we do initial setup for some shared resources at the class
level because they take a lot of time. So then the resources are just shared for
all the test methods. Testrepository just takes the list of tests that it got
from subunit partitions them up and then sends that to the testrunner. So for
each test setupClass gets run independently which significantly increases the
run time.

I have some ideas to get around this, see the thread on testing fixtures in
tempest: http://lists.openstack.org/pipermail/openstack-qa/2013-June/000517.html

I think that at this point the quickest path to success is going to be changing
how testrepository partitions tests. I've started working on doing this by adding
a new flag to partition on testCases instead of individual tests. This way we
should only be running setupClass once and hopefully give us the performance we
were expecting.


-Matt Treinish

_______________________________________________
openstack-qa mailing list
openstack-qa at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa