Open Stack

On 03/30/2015 10:44 AM, Matthew Treinish wrote:
> On Mon, Mar 30, 2015 at 12:21:18PM +0530, Rohan Kanade wrote:
>> Since tests can now be removed from Tempest <
>> https://wiki.openstack.org/wiki/QA/Tempest-test-removal> and migrated to
>> their specific projects.
>>
>> Does Tempest plan to discover/run these tests in tempest gates? If yes, how
>> is that going to be done?  Will there be a discovery mechanism in Tempest
>> to discover tests from individual projects?
>>
> No, the idea behind that wiki page is to outline the procedure for finding
> something that is out of scope and doesn't belong in tempest and is also safe
> to remove from the tempest jobs. The point of going through that entire
> procedure is that the test being removed should not be run in the tempest gates
> anymore and will become the domain of the other project.
>
> Also, IMO the moved test ideally won't be in the same pattern of a tempest test
> or have the same constraints of a tempest test and would ideally be more coupled
> to the project under test's internals. So that wouldn't be appropriate to
> include in a tempest run either.
>
> For example, the first test we removed with that procedure was:
>
> https://review.openstack.org/#/c/158852/
>
> which removed the flavor negative tests from tempest. These were just testing
> operations that would go no deeper than Nova's DB layer. Which was something
> we couldn't verify in tempest. They also didn't really belong in tempest because
> they were just implicitly verifying Nova's DB layer through API responses. The
> replacement tests:
>
> http://git.openstack.org/cgit/openstack/nova/tree/nova/tests/functional/wsgi/test_flavor_manage.py
>
> were able to verify the state of the DB was correct and ensure the correct
> behavior both in the api and nova's internals. This kind of testing is something
> which doesn't belong in tempest or any other external test suite. It is also
> what I feel we should be targeting for with project specific in-tree functional
> testing and the kind of thing we should be using the removal process on that
> wiki page for.
>
>
> -Matt Treinish
>
>
Matt, while everything you say here is true, I don't think it answers 
the whole question. neutron is also planning to move the tempest 
networking tests into the neutron repo with safeguards to prevent 
incompatible changes, but also keeping the tests in a form that is not 
so different from tempest.

The problem is that deployers/users/refstack/etc. (let's call them 
verifiers) want an "OpenStack functional verification suite". Until now 
that has been easy since most of what that requires is in tempest, and 
Rally calls tempest. But to a verifier, the fact that all the tests used 
for verification are in one tempest repo is an implementation detail. 
OpenStack verifiers do not want to lose neutron tests because they moved 
out of tempest. So verifiers will need to do something about this and it 
would be better if we all did it as a community by agreeing on a UX and 
method for locating and running all the tests that should be included in 
an OpenStack functional test suite. Even now, there are tests that are 
useful for verification that are not in tempest.

I think the answer that Boris gave 
http://lists.openstack.org/pipermail/openstack-dev/2015-March/060173.html is 
trying to address this by saying that Rally will take on the role of 
being the "OpenStack verification suite" (including performance tests). 
I don't know if that is the best answer and tempest/rally could agree on 
a UX/discovery/import mechanism, but I think we are looking at one of 
those two choices.

  -David