Open Stack

On Fri, Mar 18, 2016 at 9:06 AM, Ken'ichi Ohmichi <ken1ohmichi at gmail.com> wrote:
> 2016-03-17 4:05 GMT-07:00 Andrea Frittoli <andrea.frittoli at gmail.com>:
>> On Thu, Mar 17, 2016 at 2:57 AM Ken'ichi Ohmichi <ken1ohmichi at gmail.com>
>> wrote:
>>>
>>> 2016-03-16 19:41 GMT-07:00 Jim Rollenhagen <jim at jimrollenhagen.com>:
>>> > On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote:
>>> >> Hi
>>> >>
>>> >> I have one proposal[1] related to negative tests in Tempest, and
>>> >> hoping opinions before doing that.
>>> >>
>>> >> Now Tempest contains negative tests and sometimes patches are being
>>> >> posted for adding more negative tests, but I'd like to propose
>>> >> removing them from Tempest instead.
>>> >>
>>> >> Negative tests verify surfaces of REST APIs for each component without
>>> >> any integrations between components. That doesn't seem integration
>>> >> tests which are scope of Tempest.
>>> >> In addition, we need to spend the test operating time on different
>>> >> component's gate if adding negative tests into Tempest. For example,
>>> >> we are operating negative tests of Keystone and more
>>> >> components on the gate of Nova. That is meaningless, so we need to
>>> >> avoid more negative tests into Tempest now.
>>> >>
>>> >> If wanting to add negative tests, it is a nice option to implement
>>> >> these tests on each component repo with Tempest plugin interface. We
>>> >> can avoid operating negative tests on different component gates and
>>> >> each component team can decide what negative tests are valuable on the
>>> >> gate.
>>> >>
>>> >> In long term, all negative tests will be migrated into each component
>>> >> repo with Tempest plugin interface. We will be able to operate
>>> >> valuable negative tests only on each gate.
>>> >
>>> > So, positive tests in tempest, negative tests as a plugin.
>>> >
>>> > Is there any longer term goal to have all tests for all projects in a
>>> > plugin for that project? Seems odd to separate them.
>>>
>>> Yeah, from implementation viewpoint, that seems a little odd.
>>> but from the main scope of Tempest and to avoid unnecessary gate
>>> operation time, that can be acceptable, I feel.
>>> Negative tests can be corner cases in most cases, they don't seem
>>> integration tests.
>>
>> I think it's difficult to define a single black and white criteria for
>> negative tests, as they encompass a wide range of types of tests.
>>
>> I agree that things that only testing the API level of a service (not even a
>> DB behind) do not necessarily belong in tempest - i.e. testing of input
>> validation done by an API.  We could have a guideline for such tests to be
>> implemented as unit/functional tests in tree of the service.

Yes, this is key point here. If we see ~70% of the negative tests are
just checking API surface level (wrong input validation), which
defiantly
not belong to Tempest scope. Those should be in respective projects
repo either by functional/unit/plugin.
But in that case we have to define a very clear criteria about what
level of negative testing should be in scope of Tempest.

Also another key point is that, as we have lot of surface level
negative testing in Tempest, should we reject the new one?
For me sometime it makes difficult to reject those as we already have
some in tempest.

My vote here is we reject the new surface level negative tests and try
to move all existing negative tests(surface level) out of Tempest
ASAP.
And those can be just moved to projects functional/unit tests.

>
> Yeah, it is difficult to distinguish corner cases or not on negative
> tests as the criteria.
> If necessary to do that, we(QA team) need to read the implementation
> code of the core six services deeply during Tempest reviews. Then I
> rushed to remove all of them. My first proposal is not good according
> to feedback, but I'm happy to get feedback to see our direction :-)
>
> The guideline is a nice idea.
> If necessary to add more negative tests into Tempest, how about
> requiring to write the reason which explains new tests are not corner
> cases in the commit message?
> We can know the merit of new negative ones when reviewing.
>
>> However Tempest is also interoperability, so we should keep at least a few
>> negative API checks in tempest (for the core six services) to enforce that
>> return codes do not change inadvertently in negative cases, which could
>> break existing clients and applications.
>
> This also is a nice point.
> How to change error return codes is unclear to me at this time.
> In Nova, there are some exceptions for changing error return code
> without microversion bumping as [1]. This kind of guideline will be
> discussed later.

This makes Tempest scope little bit unclear again. If we want to
verify all error codes in Tempest then it leads to have all surface
level negative testing also in Tempest. There are lot of scenario
where error codes can be verified and will be difficult to cover all
in Tempest.

Current negative tests does not cover all error codes for all APIs. If
we try to implement all then it will be huge tests number.
I think its project which should be verifying those.

In Summary -

1. If we choose to have only valid negative tests (other than surface
level negative testing) which can verify stability/integration of API
and used by Def-core too then,
     - We should remove all existing tests which only touch surface of
APIs (wrong input validation).
2. If we want to verify error codes also in Tempest then,
     - first point becomes invalid and we need to implement all
possible error code testing.

Having mix of those always leads to have issue in reviews/development etc.

>
> Now I am fine to keep the existing negative tests in Tempest, anyway.
>
> Thanks
> Ken Ohmichi
>
> ---
> [1]: https://github.com/openstack/nova/blob/master/doc/source/api_microversion_dev.rst#f2
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev