[openstack-dev] [QA][all] Propose to remove negative tests from Tempest

Jordan Pittier jordan.pittier at scality.com
Thu Mar 17 10:22:32 UTC 2016


On Thu, Mar 17, 2016 at 2:20 AM, Ken'ichi Ohmichi <ken1ohmichi at gmail.com>

> Hi
> I have one proposal[1] related to negative tests in Tempest, and
> hoping opinions before doing that.
> Now Tempest contains negative tests and sometimes patches are being
> posted for adding more negative tests, but I'd like to propose
> removing them from Tempest instead.
> Negative tests verify surfaces of REST APIs for each component without
> any integrations between components. That doesn't seem integration
> tests which are scope of Tempest.
Tempest is not only about integration tests. I mean, we have hundreds of
tests that are not integration tests.

> In addition, we need to spend the test operating time on different
> component's gate if adding negative tests into Tempest. For example,
> we are operating negative tests of Keystone and more
> components on the gate of Nova. That is meaningless, so we need to
> avoid more negative tests into Tempest now.
You have a good point here. But this problem (running tests for project X
on project Y's gate) should be addressed more generally not only for
negative tests.

> If wanting to add negative tests, it is a nice option to implement
> these tests on each component repo with Tempest plugin interface. We
> can avoid operating negative tests on different component gates and
> each component team can decide what negative tests are valuable on the
> gate.
> In long term, all negative tests will be migrated into each component
> repo with Tempest plugin interface. We will be able to operate
> valuable negative tests only on each gate.
> Any thoughts?

I am not sure we should remove negative tests from Tempest. Agreed that we
should reject most new negative tests, but some negative
tests do test useful things imo. Also I ran all the negative tests today:
"Ran: 452 tests in 144.0000 sec." They just account for 2 minutes and 20sec
in the gate. That's very little, removing them won't bring a lot. And the
code for negative tests is quite contain, not a big maintenance burden.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160317/5019053d/attachment.html>

More information about the OpenStack-dev mailing list