Open Stack

----- Original Message -----
> From: "David Kranz" <dkranz at redhat.com>
> To: "OpenStack Development Mailing List" <openstack-dev at lists.openstack.org>
> Sent: Friday, July 25, 2014 4:54:22 PM
> Subject: [openstack-dev] [qa] The role of an abstract client in tempest
> 
> Even as a core contributor for several years, it has never been clear
> what the scope of these tests should be.
> As we move forward with the necessity of moving functional testing to
> projects, we need to answer this question for real, understanding that
> part of the mission for these tests now is validation of clouds.  Doing
> so is made difficult by the fact that the tempest api tests take a very
> opinionated view of how services are invoked. In particular, the tempest
> client is very low-level and at present the way a functional test is
> written depends on how and where it is going to run.
> 
> In an ideal world, functional tests could execute in a variety of
> environments ranging from those that completely bypass wsgi layers and
> make project api calls directly, to running in a fully integrated "real"
> environment as the tempest tests currently do. The challenge is that
> there are mismatches between how the tempest client looks to test code
> and how doing object-model api calls looks. Most of this discrepancy is
> because many pieces of invoking a service are hard-coded into the tests
> rather than being abstracted in a client. Some examples are:
> 
> 1. Response validation
> 2. json serialization/deserialization
> 3. environment description (tempest.conf)
> 4. Forced usage of addCleanUp
> 
> Maru Newby and I have proposed changing the test code to use a more
> abstract client by defining the expected signature and functionality
> of methods on the client. Roughly, the methods would take positional
> arguments for pieces that go in the url part of a REST call, and kwargs
> for the json payload. The client would take care of these enumerated
> issues (if necessary) and return an attribute dict. The test code itself
> would then just be service calls and checks of returned data. Returned
> data would be inspected as resource.id instead of resource['id']. There
> is a strawman example of this for a few neutron apis here:
> https://review.openstack.org/#/c/106916/
> Doing this would have the twin advantages of eliminating the need for
> boilerplate code in tests and making it possible to run the tests in
> different environments. It would also allow the inclusion of project
> functional tests in more general validation scenarios.
> 
> Since we are proposing to move parts of tempest into a stable library
> https://review.openstack.org/108858, we need to define the client in a
> way that meets all the needs outlined here before doing so. The actual
> work of defining the client in tempest and changing the code that uses
> it could largely be done one service at a time, in the tempest code,
> before being split out.
> 
> What do folks think about this idea?

I agree.
I also believe that streamlining the clients should take (at leaset partial)
precedence to https://blueprints.launchpad.net/tempest/+spec/tempest-client-scenarios.
I fear that in their current state, the tempest client are not ready to be used
in integration testing, and using them will push as backward on the scenario front
since we will loose much of the reusable code.

However, I believe that a week of coordinated and concentrated effort should get us
to a "good enough" state, to move on both tempest-client-scenarios bp, and further 
enhancing the tempest clients one at a time

Yair

> 
>   -David
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>