Open Stack

(Sorry for the top post.  It was this or bottom post because of company choice of email systems)

Integration tests, corner cases, negative tests.  Lots of names that don't have clear definitions in this discussion.  But, it seems like the collection of "negative tests" include both functional and integrated tests.  Maybe we should analyze the tests to see where they belong. So, perhaps we can ask an important question here:

·         Does the test provoke a change of behavior outside the designated project, depending on the output(s) from the test action?  In other words, if the test is attempting to generate a 4xx response, does it cause another project or client to act differently than if it returns a 2xx or different 4xx response?  An example of this would be the project the response is returned to attempts a retry.
This should qualify as a tempest test because it drives actions from multiple projects.  Yes, it is a "negative" api test, but run in an integrated system, it is also a normal integration test that demonstrates the integrated system behaves appropriately on expected error conditions from other projects.  Or it throws an exception or an error message and we know we have a problem with either the api or the misbehaving project.

Andrea aptly pointed out that these tests may very well be interoperability tests that should be in a gate somewhere.  After all, what are APIs, but defined interfaces between two components.  Sometimes it's between human and software, but most often in OpenStack, it's between two pieces of software written by different groups of developers.  Hence, the API *is* the integration point and ideally, all request/response options should be validated, whether happy path or not so happy.

And, oh by the way, "negative" responses are crucial interoperability points for DefCore.

--Rocky

From: Valeriy Ponomaryov March 16, 2016 11:25 PM
Main reason to split negative tests out, that I see, is attempt to remove "corner, not integrational" tests out of Tempest tree. If so, then "negative" tests is not proper choice, I think.
Tempest has lots of positive "corner" test cases. So, if move something, then we need to move exactly "corner" tests and leave only "integrational" tests of any kind - either it is negative or positive.
It is said, that doing so, all single-project related tests will be in its own repo as a plugin and only integrational tests will be in Tempest.

Valeriy

On Thu, Mar 17, 2016 at 7:31 AM, Qiming Teng <tengqim at linux.vnet.ibm.com<mailto:tengqim at linux.vnet.ibm.com>> wrote:
>
> I'd love to see this idea explored further. What happens if Tempest
> ends up without tests, as a library for shared code as well as a
> centralized place to run tests from via plugins?
>

Also curious about this. It seems weird to separate the 'positive' and
the 'negative' ones, assuming those patches are mostly contributed by
the same group of developers.

Qiming


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Kind Regards
Valeriy Ponomaryov
www.mirantis.com<http://www.mirantis.com>
vponomaryov at mirantis.com<mailto:vponomaryov at mirantis.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160317/9084e29a/attachment.html>