Open Stack

On 10/18/2013 12:04 PM, Brant Knudson wrote:
> To provide a bit more background... Keystone has a bunch of
> keystoneclient tests for the v2 API. These tests actually "git clone" a
> version of keystoneclient (master, essex-3, and 0.1.1)[0], to use for
> testing.  Maybe at some point the tests were just for client-server
> compatibility, but now they're used for more than that; for example,
> they're used for tests that require going through the paste pipeline and
> v2 controller. It's just the quickest way to get some tests written. In
> addition, there's versions of the client tests for both the kvs and sql
> backends.
>
> This causes several problems,
> 1) It looks like we're not keeping the versions to test up-to-date --
> should we checkout supported releases instead?
> 2) "git clone"ing the keystoneclient doesn't work well with parallel
> testing (we have a similar problem in our tests with our "pristine"
> database backup)

Can you go into the specifics of why?

> 3) These tests eat up lots of memory which we've gotten complaints about.

Again, can you go into the specifics of why?

> Getting v3 API keystoneclient/keystone testing in tempest is going to
> hopefully lead to getting the v2 tests out of Keystone. Thanks to Steve
> for taking this first step!
>
> For the v3 API, the tests don't use the keystoneclient but instead use
> webtest [1] and the REST API.
>
> [0]
> https://github.com/openstack/keystone/blob/master/keystone/tests/test_keystoneclient.py#L1070
> [1]
> https://github.com/openstack/keystone/blob/master/keystone/tests/test_content_types.py#L69

So v3 keystone API is one thing, but I'm a little concerned with moving 
the client testing to Tempest haphazardly.  If we are testing the API 
surface on the servers, the clients should be able to correctly test all 
of this via a mock of those API returns, which would let us separate 
concerns here and keep the client tests close to their code as unit tests.

We're actually actively trying to figure out what can migrate out of 
tempest back to the integrated projects, so that we get our biggest bang 
for our buck.

Also, realize in a tempest environment there is only going to be the 
latest version of the clients, so this is going to massively reduce your 
test environment from what you have today.

	-Sean

-- 
Sean Dague
http://dague.net