Open Stack

On Sat, Oct 19, 2013 at 07:45:33AM -0400, Sean Dague wrote:
> On 10/19/2013 04:46 AM, Steven Hardy wrote:
> >On Fri, Oct 18, 2013 at 01:48:08PM -0400, Sean Dague wrote:
> >>So v3 keystone API is one thing, but I'm a little concerned with
> >>moving the client testing to Tempest haphazardly.  If we are testing
> >>the API surface on the servers, the clients should be able to
> >>correctly test all of this via a mock of those API returns, which
> >>would let us separate concerns here and keep the client tests close
> >>to their code as unit tests.
> >
> >IME testing the API and clients separately via mocked interfaces is not
> >enough, you still end up with several categories of bugs which can (and do,
> >regularly) sneak through:
> >
> >- The API implementation is wrong, and there is a corresponding mistake in
> >   the unit tests.  This is obviously a common problem with any unit test
> >   containing mock responses, here's a recent example of such an issue,
> >   discovered while writing these tests..
> >
> >   https://review.openstack.org/#/c/51448/
> 
> Ok, fair, but was that API being tested in Tempest? (grep -r
> impersonation in tempest tree suggests not).

No, you're right, it looks like there's no API coverage of trusts at all.

> To be fair I wasn't suggesting mocking was good enough by itself, it
> was the combination of:
>  * API coverage in Tempest to ensure API contract
>  * mocking in the project based on the fact that API was solidified
> by Tempest real testing
> 
> >- The API implementation is correct, but there are undocumented or
> >   non-obvious restrictions on combinations of arguments which work, or are
> >   allowed.  This is a common issue with keystone IME, where some
> >   combinations work, and others give a 403 response with no information
> >   about why.  So you end up reading the API code or trying stuff at random,
> >   whereas with a set of client tests, we can demonstrate and verify that
> >   all required/expected usage patterns work and stay working.
> >
> >>We're actually actively trying to figure out what can migrate out of
> >>tempest back to the integrated projects, so that we get our biggest
> >>bang for our buck.
> >
> >In that case, one could argue that the best thing is to only test via the
> >clients, and move the API-surface tests back into the projects as
> >integration tests.
> 
> Here I think we'll have to disagree.
> 
> The OpenStack API is the contract. While other OpenStack components
> are using the python clients for convenience, the API should be
> equally accessible on any language. And having the API testing in
> tempest has prevented some big compatibility breaks in our API, so
> that's going to remain important.
> 
> And I think this is going to be an important part of this discussion
> because of the role that the clients have in smoothing over
> potential issues in the API.

Ok, so I'm not saying don't test the API, just that given your stated goal
of migrating stuff out of tempest, it is something which could potentially
be tested via unit/integration tests.

The problem is that none of the OpenStack projects use the API directly to
interact with keystone (or any other service for that matter), every single
one of them uses keystoneclient, so IMHO test coverage of the clients as
well as the API is really important.

> So it will be important that client library testing, if/when we add
> it, comes after the equivalent API testing being put in place.
> Ensuring API solidity is first order priority, and today there a
> huge holes in that for most projects in tempest (keystone definitely
> in that list).
> 
> >IMHO it makes sense to make the gate tests the most end-to-end tests
> >possible, using the primary integration point for each project (which in
> >most cases is the client not direct API interaction.)
> 
> Definitely agree we should have plenty of end-to-end tests in the
> gate, it's the reason we've got the scenario tests, to do exactly
> this kind of through testing.

Ok, it seems like a potential solution which may keep all involved happy
would be:
- Add new API tests which provide full coverage of the documented
  interfaces to trusts
- Add a few scenario tests which provide end-to-end testing of the most
  important interfaces (these will use the client API)

The scenario tests could just be those in my patches, moved from client_lib
to scenario/identity?

Steve