Open Stack

2016-03-16 20:27 GMT-07:00 Assaf Muller <assaf at redhat.com>:
> On Wed, Mar 16, 2016 at 10:41 PM, Jim Rollenhagen
> <jim at jimrollenhagen.com> wrote:
>> On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote:
>>> Hi
>>>
>>> I have one proposal[1] related to negative tests in Tempest, and
>>> hoping opinions before doing that.
>>>
>>> Now Tempest contains negative tests and sometimes patches are being
>>> posted for adding more negative tests, but I'd like to propose
>>> removing them from Tempest instead.
>>>
>>> Negative tests verify surfaces of REST APIs for each component without
>>> any integrations between components. That doesn't seem integration
>>> tests which are scope of Tempest.
>>> In addition, we need to spend the test operating time on different
>>> component's gate if adding negative tests into Tempest. For example,
>>> we are operating negative tests of Keystone and more
>>> components on the gate of Nova. That is meaningless, so we need to
>>> avoid more negative tests into Tempest now.
>>>
>>> If wanting to add negative tests, it is a nice option to implement
>>> these tests on each component repo with Tempest plugin interface. We
>>> can avoid operating negative tests on different component gates and
>>> each component team can decide what negative tests are valuable on the
>>> gate.
>>>
>>> In long term, all negative tests will be migrated into each component
>>> repo with Tempest plugin interface. We will be able to operate
>>> valuable negative tests only on each gate.
>>
>> So, positive tests in tempest, negative tests as a plugin.
>>
>> Is there any longer term goal to have all tests for all projects in a
>> plugin for that project? Seems odd to separate them.
>
> I'd love to see this idea explored further. What happens if Tempest
> ends up without tests, as a library for shared code as well as a
> centralized place to run tests from via plugins?

Now Tempest contains library code and the other projects can use them
as library.
We are trying to increase the library code for more usability.
The qa-spac https://review.openstack.org/#/c/275966/ is nice to be understood.

Thanks
Ken Ohmichi