Open Stack

I do think negative tests should live on, most likely in their own tempest-plugin based repository. However, I'm not sure that I agree with the reason being because they are functional tests rather than integration. If you look at the current tests, one could argue that many non-Nova tests are actually functional tests. The Nova tests have the luck of requiring other services to function at all, but volumes, identity, and other services can be and are being tested in isolation in at least some Tempest tests. If removing functional tests from Tempest is problem that is trying to be solved, then there's a scope much larger than negative tests to address.

I'm also concerned about the idea of moving these tests back to their individual project repos. To run the same tests that I do today, I would need to install each individual project as well as Tempest to get the same coverage that I get today. That feels like a quite heavy burden just to be able to keep the same coverage as there is now. This would also mean that Tempest should likely be gated on the tests that exist in the project repositories, which seems like would add a great deal of complexity and maintenance.

While I understand that some of these tests may be a duplicate effort in the context of the gate, they are needed coverage when testing a deployed OpenStack environment. Rather than sending the negative tests and others scattered back to the individual projects, would it make sense to have a separate repository to hold "extended" Tempest tests? This would allow negative tests to stay in a centralized location, as well as provide a home for tests that might not be desirable for the gate (e.x. longer scenarios, the authorization tests, etc). What I would like to avoid is the pruning of tests that are meaningful to simply reduce execution time. If what is desired is a better definition between what is Tempest and what is not, then I think that is a more important conversation to have in depth.

Daryl
________________________________________
From: Ken'ichi Ohmichi <ken1ohmichi at gmail.com>
Sent: Wednesday, March 16, 2016 8:20 PM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [QA][all] Propose to remove negative tests from        Tempest

Hi

I have one proposal[1] related to negative tests in Tempest, and
hoping opinions before doing that.

Now Tempest contains negative tests and sometimes patches are being
posted for adding more negative tests, but I'd like to propose
removing them from Tempest instead.

Negative tests verify surfaces of REST APIs for each component without
any integrations between components. That doesn't seem integration
tests which are scope of Tempest.
In addition, we need to spend the test operating time on different
component's gate if adding negative tests into Tempest. For example,
we are operating negative tests of Keystone and more
components on the gate of Nova. That is meaningless, so we need to
avoid more negative tests into Tempest now.

If wanting to add negative tests, it is a nice option to implement
these tests on each component repo with Tempest plugin interface. We
can avoid operating negative tests on different component gates and
each component team can decide what negative tests are valuable on the
gate.

In long term, all negative tests will be migrated into each component
repo with Tempest plugin interface. We will be able to operate
valuable negative tests only on each gate.

Any thoughts?

Thanks
Ken Ohmichi

---
[1]: https://review.openstack.org/#/c/293197/

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev