Open Stack

On 10/18/2013 11:59 AM, Dolph Mathews wrote:
>
> On Fri, Oct 18, 2013 at 10:34 AM, Steven Hardy <shardy at redhat.com
> <mailto:shardy at redhat.com>> wrote:
>
>     Hi all,
>
>     Starting a thread to discuss $subject, as requested in:
>
>     https://review.openstack.org/#/c/51558/
>
>     First a bit of background.  I wrote a keystoneclient patch, and ayoung
>     stated he'd like it tested via tempest before he'd ack it:
>
>     https://review.openstack.org/#/c/48462/
>
>     So I spoke to ayoung and dkranz on IRC, showing them my local tests
>     for the
>     patch.  dkranz suggested creating a "client_lib" directory, where we
>     could
>     build out a more comprehensive set of tests over time, adding to the
>     inital
>     tests related to keystone trusts client additions.
>
>     A couple of things to note:
>     - These are end-to-end tests, designed to test not only the client, but
>        also the API and keystone backend functionality.  So arguably
>     this could
>        just be a scenario test, e.g scenario/keystone/test_v3_auth.py
>
>
> I'd love to be able to run these tests against a wider variety of
> service configurations (e.g. LDAP!), which tempest is obviously more
> suitable for.

Realize that today, all the gate is a very simplistic keystone setup. If 
there had been work to bring up different keystone backends with the 
tests we currently have, I think I'd have a different take on these tests.

My main focus is how we get the biggest bang for our buck, and up until 
this point we've left direct client testing largely off the table 
because we had API testing (so the API surface should be a known 
quantity) and cli testing, because the cli turned out to be massively 
full of exceptions. But client lib testing feels like it should be able 
to be accomplished without redoing this all over again by assuming the 
contract, mocking to it, and testing in unit tests.

Is there a reason we don't think that's viable?

Also, this is probably a good summit session, so if someone wants to 
submit it, I'll land it on the QA track. Realize that if we do expand 
tempest scope to include client libs, we really need to do so somewhat 
systematically to cover all the clients, so we don't end up with just a 
few keystone client tests and that's it.

	-Sean

-- 
Sean Dague
http://dague.net