Open Stack

On Wed, Mar 16, 2016 at 10:29:48PM -0400, Adam Young wrote:
> On 03/16/2016 09:20 PM, Ken'ichi Ohmichi wrote:
> >Hi
> >
> >I have one proposal[1] related to negative tests in Tempest, and
> >hoping opinions before doing that.
> >
> >Now Tempest contains negative tests and sometimes patches are being
> >posted for adding more negative tests, but I'd like to propose
> >removing them from Tempest instead.
> >
> >Negative tests verify surfaces of REST APIs for each component without
> >any integrations between components. That doesn't seem integration
> >tests which are scope of Tempest.
> >In addition, we need to spend the test operating time on different
> >component's gate if adding negative tests into Tempest. For example,
> >we are operating negative tests of Keystone and more
> >components on the gate of Nova. That is meaningless, so we need to
> >avoid more negative tests into Tempest now.
> >
> >If wanting to add negative tests, it is a nice option to implement
> >these tests on each component repo with Tempest plugin interface. We
> >can avoid operating negative tests on different component gates and
> >each component team can decide what negative tests are valuable on the
> >gate.
> 
> Hear hear!  For Keystone, please put them in the Keystone Client Functional
> tests.

I'd recommend putting the negative tests in a plugin in the keystone
tree itself. Otherwise, how do you ensure that a change to keystone
doesn't suddenly make a bad API call go from 400 to 200, for instance?

(this assumes that keystoneclient functional tests don't run against
keystone changes, of course)

// jim