Open Stack

On 08/26/2014 10:14 AM, Zane Bitter wrote:
> Steve Baker has started the process of moving Heat tests out of the 
> Tempest repository and into the Heat repository, and we're looking for 
> some guidance on how they should be packaged in a consistent way. 
> Apparently there are a few projects already packaging functional tests 
> in the package <projectname>.tests.functional (alongside 
> <projectname>.tests.unit for the unit tests).
>
> That strikes me as odd in our context, because while the unit tests 
> run against the code in the package in which they are embedded, the 
> functional tests run against some entirely different code - whatever 
> OpenStack cloud you give it the auth URL and credentials for. So these 
> tests run from the outside, just like their ancestors in Tempest do.
>
> There's all kinds of potential confusion here for users and packagers. 
> None of it is fatal and all of it can be worked around, but if we 
> refrain from doing the thing that makes zero conceptual sense then 
> there will be no problem to work around :)
Thanks, Zane. The point of moving functional tests to projects is to be 
able to run more of them
in gate jobs for those projects, and allow tempest to survive being 
stretched-to-breaking horizontally as we scale to more projects. At the 
same time, there are benefits to the 
tempest-as-all-in-one-functional-and-integration-suite that we should 
try not to lose:

1. Strong integration testing without thinking too hard about the actual 
dependencies
2. Protection from mistaken or unwise api changes (tempest two-step 
required)
3. Exportability as a complete blackbox functional test suite that can 
be used by Rally, RefStack, deployment validation, etc.

I think (1) may be the most challenging because tests that are moved out 
of tempest might be testing some integration that is not being covered
by a scenario. We will need to make sure that tempest actually has a 
complete enough set of tests to validate integration. Even if this is 
all implemented in a way where tempest can see in-project tests as 
"plugins", there will still not be time to run them all as part of 
tempest on every commit to every project, so a selection will have to be 
made.

(2) is quite difficult. In Atlanta we talked about taking a copy of 
functional tests into tempest for stable apis. I don't know how workable 
that is but don't see any other real options except vigilance in reviews 
of patches that change functional tests.

(3) is what Zane was addressing. The in-project functional tests need to 
be written in a way that they can, at least in some configuration, run 
against a real cloud.


>
> I suspect from reading the previous thread about "In-tree functional 
> test vision" that we may actually be dealing with three categories of 
> test here rather than two:
>
> * Unit tests that run against the package they are embedded in
> * Functional tests that run against the package they are embedded in
> * Integration tests that run against a specified cloud
>
> i.e. the tests we are now trying to add to Heat might be qualitatively 
> different from the <projectname>.tests.functional suites that already 
> exist in a few projects. Perhaps someone from Neutron and/or Swift can 
> confirm?
That seems right, except that I would call the third "functional tests" 
and not "integration tests", because the purpose is not really 
integration but deep testing of a particular service. Tempest would 
continue to focus on integration testing. Is there some controversy 
about that?
The second category could include whitebox tests.

I don't know about swift, but in neutron the intent was to have these 
tests be configurable to run against a real cloud, or not. Maru Newby 
would have details.
>
> I'd like to propose that tests of the third type get their own 
> top-level package with a name of the form 
> <projectname>-integrationtests (second choice: <projectname>-tempest 
> on the principle that they're essentially plugins for Tempest). How 
> would people feel about standardising that across OpenStack?
+1 But I would not call it "integrationtests" for the reason given above.

  -David
>
> thanks,
> Zane.
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev