[openstack-qa] Priolity brainstorming my initial toughts

Attila Fazekas afazekas at redhat.com
Mon Jun 17 09:11:48 UTC 2013

The setUpClass adding multiple new variables to the name space
Which does not seams harmful, and usually helpful,  because they
avoid code repetition. (The namespace can be cleaned up, or new class instance could be created)

The test_ methods supposed to work without the help of another test_ methods action,
but it does not prohibits the usage of setUpClasses.

They would have some usage even if they do not share resources.

The test_ method MUST not depend on each other, but they can depend on multiple
setUp and setUpClass called in the correct order.

If you just select an individual test, the ordering is:
setUpClass -- normally/usually super call first 
setUp -- normally/usually super call first
tearDown -- normally/usually super call last
tearDownClass -- normally/usually super call last

A relatively simple fixture usage does not seams to be able to call,
 the first tearDownClass at the exact same order.
Even simpler usage can have difficulties even with the setUpClass.

It can cause an issue, it may be OK.

I do not feel this is the right time to add an additional layer to the model,
The existing one should be simplified and unified first.
cli, scenario, api, whitebox, thirdparty, xml/json v1,v2,v3,.. handled in similar, 
 less code repetitive way.

Smallest possible hack is extending the caller  (even with MonkeyPatching)
of the first setUp and tearDown caller, do do the setUpClass and tearDownClass call
at the right time.

The issue is just only the regular users does not have the power to cause parallel issues
in a bug free system. 
For example the keystone tests almost all have full admin privileges.
The capabilities of all OpenStack admin type user are continuously improving.

Not just the authentication special, the resources can have another relationship,
at the moment none of them seams to be on issue for a proper tearDown, but once 
we might need more complex logic, than calling several operations in reverse order.

How to handle the temporary not working test cases ?
* Add GLOBAL or DEFAULT section to the config
* Add a boolean option for class_based_resource sharing (selection between the old and new code path)
* Add an attribute option which can be logical formula (python eval ??, library for formulas ?)
  Add a skip decision into the attr decorator.
  (If it could be move to the runner in the future would be fine, one issue with the above, 
   simple way, it takes effect just after the setUp.)  
* Flag/attribute the test cases and test classes 
  (all attribute on the class should be implicitly "added" to the test_ mothod)
* Add the basic testr config to the tempest repo

No devstack/devstack-gate changes required if we do not need to spit to multiple jobs:
* The default configuration options should be the working ones
* tox.ini is in the tempest repo, anyone can send testr PoC in a single review

A simple addCleanUp does not always fully correct:
On the OK code path, it might try do delete something what is already deleted.
If the logic accepts, the already deleted resources are just deleted, 
 we might not catch a bug.
As I remember, testr provides a tagged test responses.
Can it make this type of issue at least visible ?

----- Original Message -----
From: "Christopher Yeoh" <cyeoh at au1.ibm.com>
To: "All Things QA." <openstack-qa at lists.openstack.org>
Cc: afazekas at redhat.com, mtreinish at kortar.org
Sent: Sunday, June 16, 2013 9:15:05 AM
Subject: Re: [openstack-qa] Priolity brainstorming my initial toughts

On Sat, 15 Jun 2013 01:07:25 -0400 (EDT)
Attila Fazekas <afazekas at redhat.com> wrote:
> OptimisingTestSuite seams like the suite which needed to mange the
>  resources over classes.
> Can it be used together ?

I believe OptimisingTestSuite is the long term way to go. But when
using OptimisingTestSuite setUpClass is no longer run. We only sort of
accidentally get setUpClass support when using testr at the moment
because of the way that test discovery works. testrepository does not
support setUpClass. To use OptimisingTestSuite we will first need to
convert setupClass functionality to fixtures.

yeohc at au1.ibm.com

More information about the openstack-qa mailing list