Open Stack

2016-03-17 15:05 GMT-07:00 Rochelle Grober <rochelle.grober at huawei.com>:
> (Sorry for the top post.  It was this or bottom post because of company
> choice of email systems)

(no problem :)

> Integration tests, corner cases, negative tests.  Lots of names that don't
> have clear definitions in this discussion.  But, it seems like the
> collection of "negative tests" include both functional and integrated tests.
> Maybe we should analyze the tests to see where they belong. So, perhaps we
> can ask an important question here:
>
> ·         Does the test provoke a change of behavior outside the designated
> project, depending on the output(s) from the test action?  In other words,
> if the test is attempting to generate a 4xx response, does it cause another
> project or client to act differently than if it returns a 2xx or different
> 4xx response?  An example of this would be the project the response is
> returned to attempts a retry.
>
> This should qualify as a tempest test because it drives actions from
> multiple projects.  Yes, it is a "negative" api test, but run in an
> integrated system, it is also a normal integration test that demonstrates
> the integrated system behaves appropriately on expected error conditions
> from other projects.  Or it throws an exception or an error message and we
> know we have a problem with either the api or the misbehaving project.
>
> Andrea aptly pointed out that these tests may very well be interoperability
> tests that should be in a gate somewhere.  After all, what are APIs, but
> defined interfaces between two components.  Sometimes it's between human and
> software, but most often in OpenStack, it's between two pieces of software
> written by different groups of developers.  Hence, the API *is* the
> integration point and ideally, all request/response options should be
> validated, whether happy path or not so happy.
>
> And, oh by the way, "negative" responses are crucial interoperability points
> for DefCore.

Thanks for your comment from the DefCore viewpoint.
That is very appreciated for me.

I understand the *success* status code should be stable for OpenStack
interoperability, but I don't yet for the *error* status code for
that.

I saw some *error* status code changes with valid implementations.
For example, if a client passes invalid format id as the URI to the
corresponding server, we can imagine two error status codes:
HTTP400(BadRequest) or HTTP404(NotFound).
>From the implementation viewpoint, HTTP404 can be acceptable because
such resource must not exist in its database and we can keep the
implementation simple without a format check code. However, sometimes
we want to add the format check to avoid querying its database. In
this case, the error status code is changed from HTTP404 to HTTP400.
Now Tempest is checking such error status codes as negative tests, and
the above changes would be blocked.

Maybe the above cases would be corner, but I'd like to know what is
actual merit of error status code check from the interoperability
thing.

Thanks