Open Stack

On Fri, Oct 18, 2013 at 01:48:08PM -0400, Sean Dague wrote:
> So v3 keystone API is one thing, but I'm a little concerned with
> moving the client testing to Tempest haphazardly.  If we are testing
> the API surface on the servers, the clients should be able to
> correctly test all of this via a mock of those API returns, which
> would let us separate concerns here and keep the client tests close
> to their code as unit tests.

IME testing the API and clients separately via mocked interfaces is not
enough, you still end up with several categories of bugs which can (and do,
regularly) sneak through:

- The API implementation is wrong, and there is a corresponding mistake in
  the unit tests.  This is obviously a common problem with any unit test
  containing mock responses, here's a recent example of such an issue,
  discovered while writing these tests..

  https://review.openstack.org/#/c/51448/

- The API implementation is correct, but there are undocumented or
  non-obvious restrictions on combinations of arguments which work, or are
  allowed.  This is a common issue with keystone IME, where some
  combinations work, and others give a 403 response with no information
  about why.  So you end up reading the API code or trying stuff at random,
  whereas with a set of client tests, we can demonstrate and verify that
  all required/expected usage patterns work and stay working.

> We're actually actively trying to figure out what can migrate out of
> tempest back to the integrated projects, so that we get our biggest
> bang for our buck.

In that case, one could argue that the best thing is to only test via the
clients, and move the API-surface tests back into the projects as
integration tests.

IMHO it makes sense to make the gate tests the most end-to-end tests
possible, using the primary integration point for each project (which in
most cases is the client not direct API interaction.)

Thanks for all the discussion so far, I'll make sure I attend the summit
session so we can figure out the next steps.

Also I apologize if my patches caught you off-guard, I (mis)interpreted my
initial chat with dkranz as blessing to proceed, and with Havana deadlines
looming I just hacked out the tests in an effort to get my patch into
keystoneclient.

Thanks,

Steve