Open Stack

On 10/18/2013 12:04 PM, Brant Knudson wrote:
> To provide a bit more background... Keystone has a bunch of 
> keystoneclient tests for the v2 API. These tests actually "git clone" 
> a version of keystoneclient (master, essex-3, and 0.1.1)[0], to use 
> for testing.  Maybe at some point the tests were just for 
> client-server compatibility, but now they're used for more than that; 
> for example, they're used for tests that require going through the 
> paste pipeline and v2 controller. It's just the quickest way to get 
> some tests written. In addition, there's versions of the client tests 
> for both the kvs and sql backends.
>
> This causes several problems,
> 1) It looks like we're not keeping the versions to test up-to-date -- 
> should we checkout supported releases instead?
> 2) "git clone"ing the keystoneclient doesn't work well with parallel 
> testing (we have a similar problem in our tests with our "pristine" 
> database backup)
> 3) These tests eat up lots of memory which we've gotten complaints about.
>
> Getting v3 API keystoneclient/keystone testing in tempest is going to 
> hopefully lead to getting the v2 tests out of Keystone. Thanks to 
> Steve for taking this first step!
I was going to try an approach where we used tempest to just call the 
code in Keystone as a first step.  That was one of the reasons that I 
was in favor of moving the Keystone tests into the keystone namespace.

We need to skip the git clone, step, obviously, and I am not certain 
about our use of fixtures:  we might need to redo these so the sample 
data doesn't conflict with what Tempest expects.



>
> For the v3 API, the tests don't use the keystoneclient but instead use 
> webtest [1] and the REST API.
>
> [0] 
> https://github.com/openstack/keystone/blob/master/keystone/tests/test_keystoneclient.py#L1070
> [1] 
> https://github.com/openstack/keystone/blob/master/keystone/tests/test_content_types.py#L69

We'll need V3 client support eventually, and we should use Tempest as 
the primary test environement  for that.

>
> - Brant
>
>
>
> On Fri, Oct 18, 2013 at 10:59 AM, Dolph Mathews 
> <dolph.mathews at gmail.com <mailto:dolph.mathews at gmail.com>> wrote:
>
>
>     On Fri, Oct 18, 2013 at 10:34 AM, Steven Hardy <shardy at redhat.com
>     <mailto:shardy at redhat.com>> wrote:
>
>         Hi all,
>
>         Starting a thread to discuss $subject, as requested in:
>
>         https://review.openstack.org/#/c/51558/
>
>         First a bit of background.  I wrote a keystoneclient patch,
>         and ayoung
>         stated he'd like it tested via tempest before he'd ack it:
>
>         https://review.openstack.org/#/c/48462/
>
>         So I spoke to ayoung and dkranz on IRC, showing them my local
>         tests for the
>         patch.  dkranz suggested creating a "client_lib" directory,
>         where we could
>         build out a more comprehensive set of tests over time, adding
>         to the inital
>         tests related to keystone trusts client additions.
>
>         A couple of things to note:
>         - These are end-to-end tests, designed to test not only the
>         client, but
>           also the API and keystone backend functionality.  So
>         arguably this could
>           just be a scenario test, e.g scenario/keystone/test_v3_auth.py
>
>
>     I'd love to be able to run these tests against a wider variety of
>     service configurations (e.g. LDAP!), which tempest is obviously
>     more suitable for.
>
>
>         - The intention is to excercise logic which is hard to fully
>         test with
>           unit or integration tests, and to catch issues like
>         incompatibility
>           between client and API - e.g keystoneclient tests may pass,
>         but we need
>           to make sure the client actually works against the real
>         keystone API.
>
>
>     All of our tests under keystone.tests.test_keystoneclient fall
>     into this category as well:
>
>     https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient.py
>
>     https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient_sql.py
>
>
>         Working on Heat has given me a pretty good insight into the
>         python-*client
>         API's, as we use them to orchestrate actions with every
>         openstack service;
>         IMO anything we can do to make these interfaces more robust
>         (and catch
>         bugs, several of which I found already while writing these
>         tests) is a
>         good-thing (tm).
>
>
>     ++
>
>
>         I'd welcome feedback on the patch above, and what will be the most
>         acceptable approach to the tempest team for adding these tests.
>
>         More links:
>
>         https://review.openstack.org/#/c/51559/
>         https://review.openstack.org/#/c/51560/
>         https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api
>
>         Thanks!
>
>         Steve
>
>         _______________________________________________
>         OpenStack-dev mailing list
>         OpenStack-dev at lists.openstack.org
>         <mailto:OpenStack-dev at lists.openstack.org>
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
>     -- 
>
>     -Dolph
>
>     _______________________________________________
>     OpenStack-dev mailing list
>     OpenStack-dev at lists.openstack.org
>     <mailto:OpenStack-dev at lists.openstack.org>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131018/d1faec9f/attachment.html>