Open Stack

Most negative test supposed to be very simple and we should not spend
too much time in them.

The right question:
Are we able to run 100 negative test/sec ?
Where is the time spent ?

If we are able to solve the main issue,
probably we do not need to worry about how many negative test we have.

Not all negative test a simple dumb thing.
If we would do smarter test selection, likely
we would need to keep only the slower ones.
So at the and almost nothing gained.


BTW, we can increase the number of tempest workers.

Best Regards,
Attila

----- Original Message -----
> From: "Ken'ichi Ohmichi" <ken1ohmichi at gmail.com>
> To: "OpenStack Development Mailing List" <openstack-dev at lists.openstack.org>
> Sent: Thursday, March 17, 2016 2:20:11 AM
> Subject: [openstack-dev] [QA][all] Propose to remove negative tests from	Tempest
> 
> Hi
> 
> I have one proposal[1] related to negative tests in Tempest, and
> hoping opinions before doing that.
> 
> Now Tempest contains negative tests and sometimes patches are being
> posted for adding more negative tests, but I'd like to propose
> removing them from Tempest instead.
> 
> Negative tests verify surfaces of REST APIs for each component without
> any integrations between components. That doesn't seem integration
> tests which are scope of Tempest.
> In addition, we need to spend the test operating time on different
> component's gate if adding negative tests into Tempest. For example,
> we are operating negative tests of Keystone and more
> components on the gate of Nova. That is meaningless, so we need to
> avoid more negative tests into Tempest now.
> 
> If wanting to add negative tests, it is a nice option to implement
> these tests on each component repo with Tempest plugin interface. We
> can avoid operating negative tests on different component gates and
> each component team can decide what negative tests are valuable on the
> gate.
> 
> In long term, all negative tests will be migrated into each component
> repo with Tempest plugin interface. We will be able to operate
> valuable negative tests only on each gate.
> 
> Any thoughts?
> 
> Thanks
> Ken Ohmichi
> 
> ---
> [1]: https://review.openstack.org/#/c/293197/
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>