<div dir="ltr"><div>To provide a bit more background... Keystone has a bunch of keystoneclient tests for the v2 API. These tests actually "git clone" a version of keystoneclient (master, essex-3, and 0.1.1)[0], to use for testing. Maybe at some point the tests were just for client-server compatibility, but now they're used for more than that; for example, they're used for tests that require going through the paste pipeline and v2 controller. It's just the quickest way to get some tests written. In addition, there's versions of the client tests for both the kvs and sql backends.<br>
<br>This causes several problems,<br>1) It looks like we're not keeping the versions to test up-to-date -- should we checkout supported releases instead?<br>2) "git clone"ing the keystoneclient doesn't work well with parallel testing (we have a similar problem in our tests with our "pristine" database backup)<br>
3) These tests eat up lots of memory which we've gotten complaints about.<br><br>Getting v3 API keystoneclient/keystone testing in tempest is going to hopefully lead to getting the v2 tests out of Keystone. Thanks to Steve for taking this first step!<br>
<br>For the v3 API, the tests don't use the keystoneclient but instead use webtest [1] and the REST API.<br><br>[0] <a href="https://github.com/openstack/keystone/blob/master/keystone/tests/test_keystoneclient.py#L1070">https://github.com/openstack/keystone/blob/master/keystone/tests/test_keystoneclient.py#L1070</a><br>
[1] <a href="https://github.com/openstack/keystone/blob/master/keystone/tests/test_content_types.py#L69">https://github.com/openstack/keystone/blob/master/keystone/tests/test_content_types.py#L69</a><br><br></div>- Brant<br>
<div><div><br></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Oct 18, 2013 at 10:59 AM, Dolph Mathews <span dir="ltr"><<a href="mailto:dolph.mathews@gmail.com" target="_blank">dolph.mathews@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><div class="gmail_quote"><div class="im">On Fri, Oct 18, 2013 at 10:34 AM, Steven Hardy <span dir="ltr"><<a href="mailto:shardy@redhat.com" target="_blank">shardy@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi all,<br>
<br>
Starting a thread to discuss $subject, as requested in:<br>
<br>
<a href="https://review.openstack.org/#/c/51558/" target="_blank">https://review.openstack.org/#/c/51558/</a><br>
<br>
First a bit of background. I wrote a keystoneclient patch, and ayoung<br>
stated he'd like it tested via tempest before he'd ack it:<br>
<br>
<a href="https://review.openstack.org/#/c/48462/" target="_blank">https://review.openstack.org/#/c/48462/</a><br>
<br>
So I spoke to ayoung and dkranz on IRC, showing them my local tests for the<br>
patch. dkranz suggested creating a "client_lib" directory, where we could<br>
build out a more comprehensive set of tests over time, adding to the inital<br>
tests related to keystone trusts client additions.<br>
<br>
A couple of things to note:<br>
- These are end-to-end tests, designed to test not only the client, but<br>
also the API and keystone backend functionality. So arguably this could<br>
just be a scenario test, e.g scenario/keystone/test_v3_auth.py<br></blockquote><div><br></div></div><div>I'd love to be able to run these tests against a wider variety of service configurations (e.g. LDAP!), which tempest is obviously more suitable for.</div>
<div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
- The intention is to excercise logic which is hard to fully test with<br>
unit or integration tests, and to catch issues like incompatibility<br>
between client and API - e.g keystoneclient tests may pass, but we need<br>
to make sure the client actually works against the real keystone API.<br></blockquote><div><br></div></div><div>All of our tests under keystone.tests.test_keystoneclient fall into this category as well:</div><div><br></div>
<div>
<a href="https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient.py" target="_blank">https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient.py</a></div>
<div><br></div><div> <a href="https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient_sql.py" target="_blank">https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient_sql.py</a></div>
<div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Working on Heat has given me a pretty good insight into the python-*client<br>
API's, as we use them to orchestrate actions with every openstack service;<br>
IMO anything we can do to make these interfaces more robust (and catch<br>
bugs, several of which I found already while writing these tests) is a<br>
good-thing (tm).<br></blockquote><div><br></div></div><div>++</div><div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
I'd welcome feedback on the patch above, and what will be the most<br>
acceptable approach to the tempest team for adding these tests.<br>
<br>
More links:<br>
<br>
<a href="https://review.openstack.org/#/c/51559/" target="_blank">https://review.openstack.org/#/c/51559/</a><br>
<a href="https://review.openstack.org/#/c/51560/" target="_blank">https://review.openstack.org/#/c/51560/</a><br>
<a href="https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api" target="_blank">https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api</a><br>
<br>
Thanks!<br>
<br>
Steve<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div><br></div>-Dolph
</font></span></div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>