Open Stack

++

The ugly monkey patch approach is still working fine for my downstream 
testing, but that's something I'd be happy to get rid of.

Something that may be worth considering is to have an abstraction layer on top 
of tempest clients, to allow switching the actual implementation below:

- REST call as now for the gate  jobs
- python calls for running the tests in non-integrated environments - these 
would live in-tree with the services rather than in tempest - similar  to what 
the neutron team is doing to run tests in tree
- python calls to the official clients, so that a tempest run could still be 
used to verify the python bindings  in a dedicated job

andrea

-----Original Message-----
From: Sean Dague [mailto:sean at dague.net]
Sent: 10 July 2014 12:23
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [qa] [rfc] move scenario tests to tempest client

As I've been staring at failures in the gate a lot over the past month, we've 
managed to increasingly tune the tempest client for readability and 
debugability. So when something fails in an API test, pin pointing it's 
failure point is getting easier. The scenario tests... not so much.

Using the official clients in the scenario tests was originally thought of as 
a way to get some extra testing on those clients through Tempest.
However it has a ton of debt associated with it. And I think that client 
testing should be done as functional tests in the client trees[1], not as a 
side effect of Tempest.

 * It makes the output of a fail path radically different between the 2 types
 * It adds a bunch of complexity on tenant isolation (and basic duplication 
between building accounts for both clients)
 * It generates a whole bunch of complexity around "waiting for"
resources, and safe creates which garbage collect. All of which has to be done 
above the client level because the official clients don't provide that 
functionality.

In addition the official clients don't do the right thing when hitting API 
rate limits, so are dubious in running on real clouds. There was a proposed 
ugly monkey patch approach which was just too much for us to deal with.

Migrating to tempest clients I think would clean up a ton of complexity, and 
provide for a more straight forward debuggable experience when using Tempest.

I'd like to take a temperature on this though, so comments welcomed.

	-Sean

[1] -
http://lists.openstack.org/pipermail/openstack-dev/2014-July/039733.html
(see New Thinking about our validation layers)

--
Sean Dague
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6199 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140710/f9da6ee9/attachment.bin>