Open Stack

Even as a core contributor for several years, it has never been clear 
what the scope of these tests should be.
As we move forward with the necessity of moving functional testing to 
projects, we need to answer this question for real, understanding that 
part of the mission for these tests now is validation of clouds.  Doing 
so is made difficult by the fact that the tempest api tests take a very 
opinionated view of how services are invoked. In particular, the tempest 
client is very low-level and at present the way a functional test is 
written depends on how and where it is going to run.

In an ideal world, functional tests could execute in a variety of 
environments ranging from those that completely bypass wsgi layers and 
make project api calls directly, to running in a fully integrated "real" 
environment as the tempest tests currently do. The challenge is that 
there are mismatches between how the tempest client looks to test code 
and how doing object-model api calls looks. Most of this discrepancy is 
because many pieces of invoking a service are hard-coded into the tests 
rather than being abstracted in a client. Some examples are:

1. Response validation
2. json serialization/deserialization
3. environment description (tempest.conf)
4. Forced usage of addCleanUp

Maru Newby and I have proposed changing the test code to use a more 
abstract client by defining the expected signature and functionality
of methods on the client. Roughly, the methods would take positional 
arguments for pieces that go in the url part of a REST call, and kwargs 
for the json payload. The client would take care of these enumerated 
issues (if necessary) and return an attribute dict. The test code itself 
would then just be service calls and checks of returned data. Returned 
data would be inspected as resource.id instead of resource['id']. There 
is a strawman example of this for a few neutron apis here: 
https://review.openstack.org/#/c/106916/
Doing this would have the twin advantages of eliminating the need for 
boilerplate code in tests and making it possible to run the tests in 
different environments. It would also allow the inclusion of project 
functional tests in more general validation scenarios.

Since we are proposing to move parts of tempest into a stable library 
https://review.openstack.org/108858, we need to define the client in a 
way that meets all the needs outlined here before doing so. The actual 
work of defining the client in tempest and changing the code that uses 
it could largely be done one service at a time, in the tempest code, 
before being split out.

What do folks think about this idea?

  -David