Open Stack

On 10/18/2013 01:54 PM, Sean Dague wrote:
> On 10/18/2013 11:59 AM, Dolph Mathews wrote:
>>
>> On Fri, Oct 18, 2013 at 10:34 AM, Steven Hardy <shardy at redhat.com
>> <mailto:shardy at redhat.com>> wrote:
>>
>>     Hi all,
>>
>>     Starting a thread to discuss $subject, as requested in:
>>
>>     https://review.openstack.org/#/c/51558/
>>
>>     First a bit of background.  I wrote a keystoneclient patch, and 
>> ayoung
>>     stated he'd like it tested via tempest before he'd ack it:
>>
>>     https://review.openstack.org/#/c/48462/
>>
>>     So I spoke to ayoung and dkranz on IRC, showing them my local tests
>>     for the
>>     patch.  dkranz suggested creating a "client_lib" directory, where we
>>     could
>>     build out a more comprehensive set of tests over time, adding to the
>>     inital
>>     tests related to keystone trusts client additions.
>>
>>     A couple of things to note:
>>     - These are end-to-end tests, designed to test not only the 
>> client, but
>>        also the API and keystone backend functionality.  So arguably
>>     this could
>>        just be a scenario test, e.g scenario/keystone/test_v3_auth.py
>>
>>
>> I'd love to be able to run these tests against a wider variety of
>> service configurations (e.g. LDAP!), which tempest is obviously more
>> suitable for.
>
> Realize that today, all the gate is a very simplistic keystone setup. 
> If there had been work to bring up different keystone backends with 
> the tests we currently have, I think I'd have a different take on 
> these tests.
>
> My main focus is how we get the biggest bang for our buck, and up 
> until this point we've left direct client testing largely off the 
> table because we had API testing (so the API surface should be a known 
> quantity) and cli testing, because the cli turned out to be massively 
> full of exceptions. But client lib testing feels like it should be 
> able to be accomplished without redoing this all over again by 
> assuming the contract, mocking to it, and testing in unit tests.
>
> Is there a reason we don't think that's viable?
>
> Also, this is probably a good summit session, so if someone wants to 
> submit it, I'll land it on the QA track. Realize that if we do expand 
> tempest scope to include client libs, we really need to do so somewhat 
> systematically to cover all the clients, so we don't end up with just 
> a few keystone client tests and that's it.

Client work needs to be in a project external to both the client and the 
server.  Hence our push toward Tempest.  I think that all of the 
projects have this same limitation:  In order to get changes tested in 
their client, they need a live server, but the client tests don't run a 
server.  We can't put the tests in the server, because the corresponding 
code changes have not been yet checked in to the client.  Chicken and Egg.

Client code is public code.  In many cases, it is the primary way that 
people integrate with Keystone, Nova, and other long established 
projects.  We cannot afford to break the contract in the Clients as that 
breaks things for lots of consumers.  Tempest is the system of record.  
Having Tempest devs review tests keep Keystone (and other) devs 
honest.   No more sneaking through a unit test change with code changes 
that secretly break things.  Now, if you want to change a public API,  
you need to address that in a Tempest test before ever making a change 
to either Keystone or the Client. This is like double book accounting, 
and it is a good-thing (tm).

I don't know, however, if we need to push all of the projects to do 
this, or if, intstead, we should just let Keystone show the way. Once 
the pattern is established, and we have worked out the kinks between 
Tempest and Keystone, the Keystone devs can act as amabassadors to other 
projects in order to pass on the accumulated wisdom.

There are a lot of short comings to the current testing.  Keystone runs 
SQL migration tests only against SQLite.  This is a waste of time.  In 
addition, the gate does not run them against MySQL or PostGRES,  which 
means that bugs get through.  The API tests are not run against multiple 
Backends.  At a minimum, we need to ensure that both SQL and LDAP are 
tested for Identity, and SQL and Memcached are tested for Tokens.  
Again, ideally against a real database, not just SQLite.

While we are submitting these tests to Tempest, they don't need to be 
run for commits on other projects.  Only for commits to Keystone and 
Keystone Client.  So the impact on the Gate jobs should be minimal.  
Keystone patches are not so proligfic that they alone are killing the 
gate, are they?

I think we can work toward a setup like this:

Devstack + mysql as the default.
Deploy a non-default domain via LDAP

Same thing for Postgtesql
The non-default domain in LDAP will require us sorting out some issues 
we've identified, so I would not expect that to be ready until around 
Icehouse 1 timeframe.



>
>     -Sean
>