[openstack-dev] [QA][all] Propose to remove negative tests from Tempest

Ken'ichi Ohmichi ken1ohmichi at gmail.com
Thu Mar 17 02:56:41 UTC 2016

2016-03-16 19:41 GMT-07:00 Jim Rollenhagen <jim at jimrollenhagen.com>:
> On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote:
>> Hi
>> I have one proposal[1] related to negative tests in Tempest, and
>> hoping opinions before doing that.
>> Now Tempest contains negative tests and sometimes patches are being
>> posted for adding more negative tests, but I'd like to propose
>> removing them from Tempest instead.
>> Negative tests verify surfaces of REST APIs for each component without
>> any integrations between components. That doesn't seem integration
>> tests which are scope of Tempest.
>> In addition, we need to spend the test operating time on different
>> component's gate if adding negative tests into Tempest. For example,
>> we are operating negative tests of Keystone and more
>> components on the gate of Nova. That is meaningless, so we need to
>> avoid more negative tests into Tempest now.
>> If wanting to add negative tests, it is a nice option to implement
>> these tests on each component repo with Tempest plugin interface. We
>> can avoid operating negative tests on different component gates and
>> each component team can decide what negative tests are valuable on the
>> gate.
>> In long term, all negative tests will be migrated into each component
>> repo with Tempest plugin interface. We will be able to operate
>> valuable negative tests only on each gate.
> So, positive tests in tempest, negative tests as a plugin.
> Is there any longer term goal to have all tests for all projects in a
> plugin for that project? Seems odd to separate them.

Yeah, from implementation viewpoint, that seems a little odd.
but from the main scope of Tempest and to avoid unnecessary gate
operation time, that can be acceptable, I feel.
Negative tests can be corner cases in most cases, they don't seem
integration tests.

Ken Ohmichi

More information about the OpenStack-dev mailing list