[openstack-dev] [QA][all] Propose to remove negative tests from Tempest

Rodrigo Duarte rodrigodsousa at gmail.com
Thu Mar 17 13:52:05 UTC 2016


Totally agree here, also, having positive/negative API tests in Tempest
helps in the API stability effort. Although the API is owned by the service
in question, it interacts with other services and making sure the API is
stable is valuable for the communication between them.

We know a recent example where a change in keystone API caused a change in
Cinder.

On Thu, Mar 17, 2016 at 8:05 AM, Andrea Frittoli <andrea.frittoli at gmail.com>
wrote:

>
>
> On Thu, Mar 17, 2016 at 2:57 AM Ken'ichi Ohmichi <ken1ohmichi at gmail.com>
> wrote:
>
>> 2016-03-16 19:41 GMT-07:00 Jim Rollenhagen <jim at jimrollenhagen.com>:
>> > On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote:
>> >> Hi
>> >>
>> >> I have one proposal[1] related to negative tests in Tempest, and
>> >> hoping opinions before doing that.
>> >>
>> >> Now Tempest contains negative tests and sometimes patches are being
>> >> posted for adding more negative tests, but I'd like to propose
>> >> removing them from Tempest instead.
>> >>
>> >> Negative tests verify surfaces of REST APIs for each component without
>> >> any integrations between components. That doesn't seem integration
>> >> tests which are scope of Tempest.
>> >> In addition, we need to spend the test operating time on different
>> >> component's gate if adding negative tests into Tempest. For example,
>> >> we are operating negative tests of Keystone and more
>> >> components on the gate of Nova. That is meaningless, so we need to
>> >> avoid more negative tests into Tempest now.
>> >>
>> >> If wanting to add negative tests, it is a nice option to implement
>> >> these tests on each component repo with Tempest plugin interface. We
>> >> can avoid operating negative tests on different component gates and
>> >> each component team can decide what negative tests are valuable on the
>> >> gate.
>> >>
>> >> In long term, all negative tests will be migrated into each component
>> >> repo with Tempest plugin interface. We will be able to operate
>> >> valuable negative tests only on each gate.
>> >
>> > So, positive tests in tempest, negative tests as a plugin.
>> >
>> > Is there any longer term goal to have all tests for all projects in a
>> > plugin for that project? Seems odd to separate them.
>>
>> Yeah, from implementation viewpoint, that seems a little odd.
>> but from the main scope of Tempest and to avoid unnecessary gate
>> operation time, that can be acceptable, I feel.
>> Negative tests can be corner cases in most cases, they don't seem
>> integration tests.
>>
>
> I think it's difficult to define a single black and white criteria for
> negative tests, as they encompass a wide range of types of tests.
>
> I agree that things that only testing the API level of a service (not even
> a DB behind) do not necessarily belong in tempest - i.e. testing of input
> validation done by an API.  We could have a guideline for such tests to be
> implemented as unit/functional tests in tree of the service.
>
> However Tempest is also interoperability, so we should keep at least a few
> negative API checks in tempest (for the core six services) to enforce that
> return codes do not change inadvertently in negative cases, which could
> break existing clients and applications.
>
> If a project was to move all negative tests out of tempest, than they
> might consider have hacking rules to prevent modifying the code and tests
> at the same time, and change behaviour inadvertently.
>
> andrea
>
>
>>
>> Thanks
>> Ken Ohmichi
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Rodrigo Duarte Sousa
Senior Quality Engineer @ Red Hat
MSc in Computer Science
http://rodrigods.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160317/c8ed9435/attachment.html>


More information about the OpenStack-dev mailing list