[Openstack] [QA] Aligning "smoke" / "acceptance" / "promotion" test efforts
Jay Pipes
jaypipes at gmail.com
Thu May 10 00:46:09 UTC 2012
On 05/03/2012 03:54 PM, Daryl Walleck wrote:
> So my first question is around this. So is the claim is that the client
> tools are the default interface for the applications?
Sorry, perhaps a better term would have been "the most common interface
to OpenStack Compute"...
> While that works
> for coders in python, what about people using other languages? Even
> then, there's no guarantee that the clients in different languages are
> implemented in the same way. Tempest was designed originally because
> while it does use an abstraction between the API and the tests, there is
> nothing to "assist" the user by retrying and the like. While I think
> there's a place for writing tests using the command line clients, to me
> that would be a smoke test of a client and not as much a smoke test of
> the API.
I understand your point, and I want to make it clear that I'm not
advocating for getting rid of the Tempest REST client at all. I'm only
advocating that we have a set of tests -- smoke tests -- that use the
Python novaclient library, run a small subset of the total tests, and
exercise only the most common and simple use cases for OpenStack.
Here is where I think a sensible breakdown is:
* "Smoke tests"
- Test ONLY *simple*, *fast* code paths for common use cases
- Use a Manager class that has a novaclient Client object
* Negative tests
- Test all code paths that should produce errors or negative results
(NotFound, etc)
- Use a Manager class that has the Tempest REST client
* Fuzz tests
- Test all code paths with randomized input data
- Use a Manager class that has the Tempest REST client
* Advanced tests
- Test extensions, interaction between multiple components, advanced
features of the API, etc
- Use a Manager class that has a novaclient Client object OR the
Tempest REST client, depending on whether:
- novaclient actually has the ability to perform the necessary actions
- If the test needs to poke at the API in a way that novaclient would
interfere with -- and thus the Tempest REST client would be ideal
Best,
-jay
More information about the Openstack
mailing list