[openstack-qa] [Tempest] Blueprint: add-scenario-tests

Attila Fazekas afazekas at redhat.com
Thu Apr 4 18:02:51 UTC 2013


If the test subject behave like working OpenStack deployment it is a
"working OpenStack" installation, this is the concept behind
tempest/tempest. (Black Box) 
The installation method is correct when the result is a "working
OpenStack".

The tests cases are focusing on functionality and API testing,
these should behave in same way regardless to the installation
method and configuration.

Some OpenStack configuration does not able to do certain operation, for
example cinder with nfs is not able to create snapshots,
but it does not makes the model invalid.  Usually just small parts
effected because of a not supported functionality.

tempest/tempest does not looks behind the scenes.
If as an "end-user" we see the deployment is working as expected, the
test subject considered good.

Two deployment's internal behavior can be very different, even if they
just differs in one configuration variable ...

But, we can use the same tempest/tempest to test them.

The gate test just using  single virtual machines for testing the system
with 3 "little" different configuration.


It would be good to see those interesting scenarios, which can not fit
to the tempest/tempest model and causing significant coverage lost.

I would like to see 3-4 scenario specification with detailed
installation/configuration steps. In order to see where can it reach
additional code paths.


In a multi-node environment we have greater chance to see certain bug
types and we will have little more behavior to test, like volume types and
scheduling.

I agree, for hunting those issue mainly we need better "stress"
framework, probably with some temporary fault injection method in
addition.

I agree we need something which test things which are outside from the
gating tempest scope.



This scenario member/role things may included policy.json settings. 

BTW: What is the reason behind these settings are in a configuration
file and not in a database ?

The policy rules interpreted by a module which is in the oslo and used
by multiple projects.
This module functionality tested by unit tests. 
Unit test are good for testing it's decision making capabilities, we do
not really need an integration test tool for these IMHO.

Every requested action should pass a policy testing before execution!
BTW: Can I say if an action needs two policy check, it is a bug ?  

BTW: Not just the policy checks are able to refuse an operation.

Every policy check should decide is the operation authorized or not.
We have relatively good coverage for the authorized code path, but the
not authorized code path, is not tested intensively.

Now, we have 3 type of user's which attempting an operation: 
- admin
- non-admin user
- non-admin user from a different tenant

Now, we do not have enough negative test case even for the possible
situations from these 3 groups.


The main issue with the policy, it can changes the expected behavior in
tempest viewpoint.

The policy.json has the ability to define very strange behaviors, so we
would need strange test framework for following all of the
possibilities.
Do we want to test all possibilities on tempest/tempest level ?
 
Tempest expect the test user (demo), has the ability (rights) to pass
the test cases.

If we start creating test users which has all necessary roles for a test
case, we do not test anything new, because the policy module itself
tested by unit tests and in the higher level we just see the very same
decision..

We are testing a new code path only if the operation rejected by the
policy module.   
  
We should change the policy settings on the gate by extending a model
with a 'nobody' users.

Nobody is user for testing the situation when the user is in the same
tenant, but does not have permission for any operation.
  
We should have "somebody" users who are not admin users, but have
permission for doing the things which was allowed by default policy.json
settings.

In this case we will have 4 type:
- admin
- "somebody"
- "nobody" (rejection caused by policy)
- "somebody" user from a different tenant

Creating a tempest/devstack migration path seams possible.

We should add some role related config parameter to tempest.conf for the run time user creation as
jaypipes mentioned since long.

We should add some role to the demo users in devstack...
We need to change all default policy.json on the gate...

Is this "model" good enough for judging the policy things ?

Do we have any other "strange" thing to test ?

Best Regards,
Attila






More information about the openstack-qa mailing list