Open Stack

On 10/18/2013 01:54 PM, Sean Dague wrote:
> On 10/18/2013 11:59 AM, Dolph Mathews wrote:
>>
>> On Fri, Oct 18, 2013 at 10:34 AM, Steven Hardy <shardy at redhat.com
>> <mailto:shardy at redhat.com>> wrote:
>>
>>     Hi all,
>>
>>     Starting a thread to discuss $subject, as requested in:
>>
>>     https://review.openstack.org/#/c/51558/
>>
>>     First a bit of background.  I wrote a keystoneclient patch, and 
>> ayoung
>>     stated he'd like it tested via tempest before he'd ack it:
>>
>>     https://review.openstack.org/#/c/48462/
>>
>>     So I spoke to ayoung and dkranz on IRC, showing them my local tests
>>     for the
>>     patch.  dkranz suggested creating a "client_lib" directory, where we
>>     could
>>     build out a more comprehensive set of tests over time, adding to the
>>     inital
>>     tests related to keystone trusts client additions.
>>
>>     A couple of things to note:
>>     - These are end-to-end tests, designed to test not only the 
>> client, but
>>        also the API and keystone backend functionality.  So arguably
>>     this could
>>        just be a scenario test, e.g scenario/keystone/test_v3_auth.py
>>
>>
>> I'd love to be able to run these tests against a wider variety of
>> service configurations (e.g. LDAP!), which tempest is obviously more
>> suitable for.
>
> Realize that today, all the gate is a very simplistic keystone setup. 
> If there had been work to bring up different keystone backends with 
> the tests we currently have, I think I'd have a different take on 
> these tests.
>
> My main focus is how we get the biggest bang for our buck, and up 
> until this point we've left direct client testing largely off the 
> table because we had API testing (so the API surface should be a known 
> quantity) and cli testing, because the cli turned out to be massively 
> full of exceptions. But client lib testing feels like it should be 
> able to be accomplished without redoing this all over again by 
> assuming the contract, mocking to it, and testing in unit tests.
I really don't understand why cli (shell programming language) and the 
python clients should be treated differently. The exact same argument 
could be made for cli.
>
> Is there a reason we don't think that's viable?
>
> Also, this is probably a good summit session, so if someone wants to 
> submit it, I'll land it on the QA track. Realize that if we do expand 
> tempest scope to include client libs, we really need to do so somewhat 
> systematically to cover all the clients, so we don't end up with just 
> a few keystone client tests and that's it.
I put this in the etherpad a few weeks ago "Strategy for avoiding 
duplication with unit tests." We need a real strategy for where 
different kinds of tests should go. And all this makes it even more 
clear that we need a way to separate the functional description of a 
test from the environment in which it can run. The decision of whether a 
test should run in a real env, or mocked in various ways, should be more 
abstracted from the actual test code where possible IMO.

  -David