<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 10/18/2013 12:04 PM, Brant Knudson
wrote:<br>
</div>
<blockquote
cite="mid:CAHjeE=Qgp8n-Jc6Zvry6U4Xin+ANj5Xt+4rKfwz5gmvNgLTXRQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>To provide a bit more background... Keystone has a bunch of
keystoneclient tests for the v2 API. These tests actually "git
clone" a version of keystoneclient (master, essex-3, and
0.1.1)[0], to use for testing. Maybe at some point the tests
were just for client-server compatibility, but now they're
used for more than that; for example, they're used for tests
that require going through the paste pipeline and v2
controller. It's just the quickest way to get some tests
written. In addition, there's versions of the client tests for
both the kvs and sql backends.<br>
<br>
This causes several problems,<br>
1) It looks like we're not keeping the versions to test
up-to-date -- should we checkout supported releases instead?<br>
2) "git clone"ing the keystoneclient doesn't work well with
parallel testing (we have a similar problem in our tests with
our "pristine" database backup)<br>
3) These tests eat up lots of memory which we've gotten
complaints about.<br>
<br>
Getting v3 API keystoneclient/keystone testing in tempest is
going to hopefully lead to getting the v2 tests out of
Keystone. Thanks to Steve for taking this first step!<br>
</div>
</div>
</blockquote>
I was going to try an approach where we used tempest to just call
the code in Keystone as a first step. That was one of the reasons
that I was in favor of moving the Keystone tests into the keystone
namespace.<br>
<br>
We need to skip the git clone, step, obviously, and I am not certain
about our use of fixtures: we might need to redo these so the
sample data doesn't conflict with what Tempest expects.<br>
<br>
<br>
<br>
<blockquote
cite="mid:CAHjeE=Qgp8n-Jc6Zvry6U4Xin+ANj5Xt+4rKfwz5gmvNgLTXRQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<br>
For the v3 API, the tests don't use the keystoneclient but
instead use webtest [1] and the REST API.<br>
<br>
[0] <a moz-do-not-send="true"
href="https://github.com/openstack/keystone/blob/master/keystone/tests/test_keystoneclient.py#L1070">https://github.com/openstack/keystone/blob/master/keystone/tests/test_keystoneclient.py#L1070</a><br>
[1] <a moz-do-not-send="true"
href="https://github.com/openstack/keystone/blob/master/keystone/tests/test_content_types.py#L69">https://github.com/openstack/keystone/blob/master/keystone/tests/test_content_types.py#L69</a><br>
</div>
</div>
</blockquote>
<br>
We'll need V3 client support eventually, and we should use Tempest
as the primary test environement for that.<br>
<br>
<blockquote
cite="mid:CAHjeE=Qgp8n-Jc6Zvry6U4Xin+ANj5Xt+4rKfwz5gmvNgLTXRQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
- Brant<br>
<div>
<div><br>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Fri, Oct 18, 2013 at 10:59 AM, Dolph
Mathews <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:dolph.mathews@gmail.com" target="_blank">dolph.mathews@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra">
<div class="gmail_quote">
<div class="im">On Fri, Oct 18, 2013 at 10:34 AM,
Steven Hardy <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:shardy@redhat.com" target="_blank">shardy@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi
all,<br>
<br>
Starting a thread to discuss $subject, as
requested in:<br>
<br>
<a moz-do-not-send="true"
href="https://review.openstack.org/#/c/51558/"
target="_blank">https://review.openstack.org/#/c/51558/</a><br>
<br>
First a bit of background. I wrote a
keystoneclient patch, and ayoung<br>
stated he'd like it tested via tempest before he'd
ack it:<br>
<br>
<a moz-do-not-send="true"
href="https://review.openstack.org/#/c/48462/"
target="_blank">https://review.openstack.org/#/c/48462/</a><br>
<br>
So I spoke to ayoung and dkranz on IRC, showing
them my local tests for the<br>
patch. dkranz suggested creating a "client_lib"
directory, where we could<br>
build out a more comprehensive set of tests over
time, adding to the inital<br>
tests related to keystone trusts client additions.<br>
<br>
A couple of things to note:<br>
- These are end-to-end tests, designed to test not
only the client, but<br>
also the API and keystone backend functionality.
So arguably this could<br>
just be a scenario test, e.g
scenario/keystone/test_v3_auth.py<br>
</blockquote>
<div><br>
</div>
</div>
<div>I'd love to be able to run these tests against a
wider variety of service configurations (e.g.
LDAP!), which tempest is obviously more suitable
for.</div>
<div class="im">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
- The intention is to excercise logic which is
hard to fully test with<br>
unit or integration tests, and to catch issues
like incompatibility<br>
between client and API - e.g keystoneclient
tests may pass, but we need<br>
to make sure the client actually works against
the real keystone API.<br>
</blockquote>
<div><br>
</div>
</div>
<div>All of our tests under
keystone.tests.test_keystoneclient fall into this
category as well:</div>
<div><br>
</div>
<div>
<a moz-do-not-send="true"
href="https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient.py"
target="_blank">https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient.py</a></div>
<div><br>
</div>
<div> <a moz-do-not-send="true"
href="https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient_sql.py"
target="_blank">https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient_sql.py</a></div>
<div class="im">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
Working on Heat has given me a pretty good insight
into the python-*client<br>
API's, as we use them to orchestrate actions with
every openstack service;<br>
IMO anything we can do to make these interfaces
more robust (and catch<br>
bugs, several of which I found already while
writing these tests) is a<br>
good-thing (tm).<br>
</blockquote>
<div><br>
</div>
</div>
<div>++</div>
<div class="im">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
I'd welcome feedback on the patch above, and what
will be the most<br>
acceptable approach to the tempest team for adding
these tests.<br>
<br>
More links:<br>
<br>
<a moz-do-not-send="true"
href="https://review.openstack.org/#/c/51559/"
target="_blank">https://review.openstack.org/#/c/51559/</a><br>
<a moz-do-not-send="true"
href="https://review.openstack.org/#/c/51560/"
target="_blank">https://review.openstack.org/#/c/51560/</a><br>
<a moz-do-not-send="true"
href="https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api"
target="_blank">https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api</a><br>
<br>
Thanks!<br>
<br>
Steve<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:OpenStack-dev@lists.openstack.org"
target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
</div>
</div>
<span class="HOEnZb"><font color="#888888"><br>
<br clear="all">
<div><br>
</div>
-- <br>
<div><br>
</div>
-Dolph
</font></span></div>
</div>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>