Open Stack

To provide a bit more background... Keystone has a bunch of keystoneclient
tests for the v2 API. These tests actually "git clone" a version of
keystoneclient (master, essex-3, and 0.1.1)[0], to use for testing.  Maybe
at some point the tests were just for client-server compatibility, but now
they're used for more than that; for example, they're used for tests that
require going through the paste pipeline and v2 controller. It's just the
quickest way to get some tests written. In addition, there's versions of
the client tests for both the kvs and sql backends.

This causes several problems,
1) It looks like we're not keeping the versions to test up-to-date --
should we checkout supported releases instead?
2) "git clone"ing the keystoneclient doesn't work well with parallel
testing (we have a similar problem in our tests with our "pristine"
database backup)
3) These tests eat up lots of memory which we've gotten complaints about.

Getting v3 API keystoneclient/keystone testing in tempest is going to
hopefully lead to getting the v2 tests out of Keystone. Thanks to Steve for
taking this first step!

For the v3 API, the tests don't use the keystoneclient but instead use
webtest [1] and the REST API.

[0]
https://github.com/openstack/keystone/blob/master/keystone/tests/test_keystoneclient.py#L1070
[1]
https://github.com/openstack/keystone/blob/master/keystone/tests/test_content_types.py#L69

- Brant



On Fri, Oct 18, 2013 at 10:59 AM, Dolph Mathews <dolph.mathews at gmail.com>wrote:

>
> On Fri, Oct 18, 2013 at 10:34 AM, Steven Hardy <shardy at redhat.com> wrote:
>
>> Hi all,
>>
>> Starting a thread to discuss $subject, as requested in:
>>
>> https://review.openstack.org/#/c/51558/
>>
>> First a bit of background.  I wrote a keystoneclient patch, and ayoung
>> stated he'd like it tested via tempest before he'd ack it:
>>
>> https://review.openstack.org/#/c/48462/
>>
>> So I spoke to ayoung and dkranz on IRC, showing them my local tests for
>> the
>> patch.  dkranz suggested creating a "client_lib" directory, where we could
>> build out a more comprehensive set of tests over time, adding to the
>> inital
>> tests related to keystone trusts client additions.
>>
>> A couple of things to note:
>> - These are end-to-end tests, designed to test not only the client, but
>>   also the API and keystone backend functionality.  So arguably this could
>>   just be a scenario test, e.g scenario/keystone/test_v3_auth.py
>>
>
> I'd love to be able to run these tests against a wider variety of service
> configurations (e.g. LDAP!), which tempest is obviously more suitable for.
>
>
>>
>> - The intention is to excercise logic which is hard to fully test with
>>   unit or integration tests, and to catch issues like incompatibility
>>   between client and API - e.g keystoneclient tests may pass, but we need
>>   to make sure the client actually works against the real keystone API.
>>
>
> All of our tests under keystone.tests.test_keystoneclient fall into this
> category as well:
>
>
> https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient.py
>
>
> https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient_sql.py
>
>
>>
>> Working on Heat has given me a pretty good insight into the python-*client
>> API's, as we use them to orchestrate actions with every openstack service;
>> IMO anything we can do to make these interfaces more robust (and catch
>> bugs, several of which I found already while writing these tests) is a
>> good-thing (tm).
>>
>
> ++
>
>
>>
>> I'd welcome feedback on the patch above, and what will be the most
>> acceptable approach to the tempest team for adding these tests.
>>
>> More links:
>>
>> https://review.openstack.org/#/c/51559/
>> https://review.openstack.org/#/c/51560/
>> https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api
>>
>> Thanks!
>>
>> Steve
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
>
> -Dolph
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131018/f02f7fc2/attachment.html>