Open Stack

On 10/19/2013 04:46 AM, Steven Hardy wrote:
> On Fri, Oct 18, 2013 at 01:48:08PM -0400, Sean Dague wrote:
>> So v3 keystone API is one thing, but I'm a little concerned with
>> moving the client testing to Tempest haphazardly.  If we are testing
>> the API surface on the servers, the clients should be able to
>> correctly test all of this via a mock of those API returns, which
>> would let us separate concerns here and keep the client tests close
>> to their code as unit tests.
>
> IME testing the API and clients separately via mocked interfaces is not
> enough, you still end up with several categories of bugs which can (and do,
> regularly) sneak through:
>
> - The API implementation is wrong, and there is a corresponding mistake in
>    the unit tests.  This is obviously a common problem with any unit test
>    containing mock responses, here's a recent example of such an issue,
>    discovered while writing these tests..
>
>    https://review.openstack.org/#/c/51448/

Ok, fair, but was that API being tested in Tempest? (grep -r 
impersonation in tempest tree suggests not).

To be fair I wasn't suggesting mocking was good enough by itself, it was 
the combination of:
  * API coverage in Tempest to ensure API contract
  * mocking in the project based on the fact that API was solidified by 
Tempest real testing

> - The API implementation is correct, but there are undocumented or
>    non-obvious restrictions on combinations of arguments which work, or are
>    allowed.  This is a common issue with keystone IME, where some
>    combinations work, and others give a 403 response with no information
>    about why.  So you end up reading the API code or trying stuff at random,
>    whereas with a set of client tests, we can demonstrate and verify that
>    all required/expected usage patterns work and stay working.
>
>> We're actually actively trying to figure out what can migrate out of
>> tempest back to the integrated projects, so that we get our biggest
>> bang for our buck.
>
> In that case, one could argue that the best thing is to only test via the
> clients, and move the API-surface tests back into the projects as
> integration tests.

Here I think we'll have to disagree.

The OpenStack API is the contract. While other OpenStack components are 
using the python clients for convenience, the API should be equally 
accessible on any language. And having the API testing in tempest has 
prevented some big compatibility breaks in our API, so that's going to 
remain important.

And I think this is going to be an important part of this discussion 
because of the role that the clients have in smoothing over potential 
issues in the API.

So it will be important that client library testing, if/when we add it, 
comes after the equivalent API testing being put in place. Ensuring API 
solidity is first order priority, and today there a huge holes in that 
for most projects in tempest (keystone definitely in that list).

> IMHO it makes sense to make the gate tests the most end-to-end tests
> possible, using the primary integration point for each project (which in
> most cases is the client not direct API interaction.)

Definitely agree we should have plenty of end-to-end tests in the gate, 
it's the reason we've got the scenario tests, to do exactly this kind of 
through testing.

> Thanks for all the discussion so far, I'll make sure I attend the summit
> session so we can figure out the next steps.
>
> Also I apologize if my patches caught you off-guard, I (mis)interpreted my
> initial chat with dkranz as blessing to proceed, and with Havana deadlines
> looming I just hacked out the tests in an effort to get my patch into
> keystoneclient.

No worries, it was just new to most of us, and we were really deliberate 
about the test directory structure and scope out of last summit, so it's 
just something I don't want to change lightly. But the discussion is 
flowing now, which is all good... and let's us figure out the best way 
forward.

	-Sean

-- 
Sean Dague
http://dague.net