<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 07/10/2014 08:08 AM, Frittoli,
Andrea (HP Cloud) wrote:<br>
</div>
<blockquote
cite="mid:A36B1DF8CF6E574F9E151E2766C8C3614B36A653@G9W0345.americas.hpqcorp.net"
type="cite">
<pre wrap="">++
The ugly monkey patch approach is still working fine for my downstream
testing, but that's something I'd be happy to get rid of.
Something that may be worth considering is to have an abstraction layer on top
of tempest clients, to allow switching the actual implementation below:
- REST call as now for the gate jobs
- python calls for running the tests in non-integrated environments - these
would live in-tree with the services rather than in tempest - similar to what
the neutron team is doing to run tests in tree
- python calls to the official clients, so that a tempest run could still be
used to verify the python bindings in a dedicated job
</pre>
</blockquote>
+1 to using tempest client. The requirement for enhanced debugging
features was not seen as critical when the original decision was
made. I don't think the badness<br>
of the current situation was anticipated.<br>
<br>
The "abstraction layer" comment is related to the discussion about
moving functional api tests to projects with a "retargetable"
client. I was discussing this with<br>
Maru yesterday at the neutron mid-cycle. In addition to abstracting
the client, we need to abstract the way the test code calls the
client and handles the results.<br>
We looked at some of the networking tests in tempest. In addition to
the boilerplate code for checking success response codes, which I
have started moving to the clients, there is also boilerplate code
around "deserializing" the body that is returned. There is also lack
of uniformity of what the various client method signatures look
like. Here is a strawman proposal we came up with to unify this:<br>
<br>
Most of our REST APIs accept some parameters that are inserted into
the url string, and others that become part of a json payload. They
return a response with a json body. A python client method should
accept arguments for each inserted parameter, and **kwargs for the
json part. If multiple success response codes might be returned, the
method would take another argument specifying which should be
checked.<br>
<br>
The client methods should no longer return (resp, body) where body
is somewhat raw. Since response checking will now be done all at the
client, the resp return value is<br>
no longer needed. The json body should be returned as a single
return value, but as an attribute dict. Any extraneous top-level
dict envelope can be stripped out. For example, the neutron "create"
apis return a top-level dict with one value and a key that is the
same name as the resource being created.<br>
<br>
Doing this would have several advantages:<br>
<br>
1. The test code would be smaller.<br>
2. The test code would only be involved with behavior checking and
any client-specific checking or serialize/deserialize would be done
by the client.<br>
3. As a result of (2), there would be sufficient abstraction that a
variety of clients could be used by the same test code.<br>
<br>
-David<br>
<blockquote
cite="mid:A36B1DF8CF6E574F9E151E2766C8C3614B36A653@G9W0345.americas.hpqcorp.net"
type="cite">
<pre wrap="">
andrea
-----Original Message-----
From: Sean Dague [<a class="moz-txt-link-freetext" href="mailto:sean@dague.net">mailto:sean@dague.net</a>]
Sent: 10 July 2014 12:23
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [qa] [rfc] move scenario tests to tempest client
As I've been staring at failures in the gate a lot over the past month, we've
managed to increasingly tune the tempest client for readability and
debugability. So when something fails in an API test, pin pointing it's
failure point is getting easier. The scenario tests... not so much.
Using the official clients in the scenario tests was originally thought of as
a way to get some extra testing on those clients through Tempest.
However it has a ton of debt associated with it. And I think that client
testing should be done as functional tests in the client trees[1], not as a
side effect of Tempest.
* It makes the output of a fail path radically different between the 2 types
* It adds a bunch of complexity on tenant isolation (and basic duplication
between building accounts for both clients)
* It generates a whole bunch of complexity around "waiting for"
resources, and safe creates which garbage collect. All of which has to be done
above the client level because the official clients don't provide that
functionality.
In addition the official clients don't do the right thing when hitting API
rate limits, so are dubious in running on real clouds. There was a proposed
ugly monkey patch approach which was just too much for us to deal with.
Migrating to tempest clients I think would clean up a ton of complexity, and
provide for a more straight forward debuggable experience when using Tempest.
I'd like to take a temperature on this though, so comments welcomed.
-Sean
[1] -
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/pipermail/openstack-dev/2014-July/039733.html">http://lists.openstack.org/pipermail/openstack-dev/2014-July/039733.html</a>
(see New Thinking about our validation layers)
--
Sean Dague
<a class="moz-txt-link-freetext" href="http://dague.net">http://dague.net</a>
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>