Open Stack

2016-03-17 5:32 GMT-07:00 Adam Young <ayoung at redhat.com>:
> On 03/16/2016 11:01 PM, Ken'ichi Ohmichi wrote:
>>
>> 2016-03-16 19:29 GMT-07:00 Adam Young <ayoung at redhat.com>:
>>>
>>> On 03/16/2016 09:20 PM, Ken'ichi Ohmichi wrote:
>>>>
>>>> Hi
>>>>
>>>> I have one proposal[1] related to negative tests in Tempest, and
>>>> hoping opinions before doing that.
>>>>
>>>> Now Tempest contains negative tests and sometimes patches are being
>>>> posted for adding more negative tests, but I'd like to propose
>>>> removing them from Tempest instead.
>>>>
>>>> Negative tests verify surfaces of REST APIs for each component without
>>>> any integrations between components. That doesn't seem integration
>>>> tests which are scope of Tempest.
>>>> In addition, we need to spend the test operating time on different
>>>> component's gate if adding negative tests into Tempest. For example,
>>>> we are operating negative tests of Keystone and more
>>>> components on the gate of Nova. That is meaningless, so we need to
>>>> avoid more negative tests into Tempest now.
>>>>
>>>> If wanting to add negative tests, it is a nice option to implement
>>>> these tests on each component repo with Tempest plugin interface. We
>>>> can avoid operating negative tests on different component gates and
>>>> each component team can decide what negative tests are valuable on the
>>>> gate.
>>>
>>>
>>> Hear hear!  For Keystone, please put them in the Keystone Client
>>> Functional
>>> tests.
>>
>> They are negative tests of Keystone API, not keystoneclient tests.
>> These tests are implemented with Tempest original rest clients without
>> official clients because it is nice to test variable cases in Tempest.
>> So it is nice to keep them in Keystone repo, I feel.
>
>
> That works, too, and there is a Functional test section in Keystone Proper.
>
> Most of the Keystone APIs are fairly well covered by the Unit tests, which
> make a full HTTP call through a liteish server, so the functional tests are
> really to cover the live database (soon to include LDAP, hopefully)
> integrations.

Nice point.
Yeah, most these cases(without DB) are covered by unit tests of each
project, I guess.
At least, Nova also covers these cases with unit tests.

Thanks