Open Stack

On 08/26/2014 10:14 AM, Zane Bitter wrote:
> Steve Baker has started the process of moving Heat tests out of the
> Tempest repository and into the Heat repository, and we're looking for
> some guidance on how they should be packaged in a consistent way.
> Apparently there are a few projects already packaging functional tests
> in the package <projectname>.tests.functional (alongside
> <projectname>.tests.unit for the unit tests).
>
> That strikes me as odd in our context, because while the unit tests run
> against the code in the package in which they are embedded, the
> functional tests run against some entirely different code - whatever
> OpenStack cloud you give it the auth URL and credentials for. So these
> tests run from the outside, just like their ancestors in Tempest do.
>
> There's all kinds of potential confusion here for users and packagers.
> None of it is fatal and all of it can be worked around, but if we
> refrain from doing the thing that makes zero conceptual sense then there
> will be no problem to work around :)
>
> I suspect from reading the previous thread about "In-tree functional
> test vision" that we may actually be dealing with three categories of
> test here rather than two:
>
> * Unit tests that run against the package they are embedded in
> * Functional tests that run against the package they are embedded in
> * Integration tests that run against a specified cloud
>
> i.e. the tests we are now trying to add to Heat might be qualitatively
> different from the <projectname>.tests.functional suites that already
> exist in a few projects. Perhaps someone from Neutron and/or Swift can
> confirm?
>
> I'd like to propose that tests of the third type get their own top-level
> package with a name of the form <projectname>-integrationtests (second
> choice: <projectname>-tempest on the principle that they're essentially
> plugins for Tempest). How would people feel about standardising that
> across OpenStack?

By its nature, Heat is one of the only projects that would have 
integration tests of this nature. For Nova, there are some "functional" 
tests in nova/tests/integrated/ (yeah, badly named, I know) that are 
tests of the REST API endpoints and running service daemons (the things 
that are RPC endpoints), with a bunch of stuff faked out (like RPC 
comms, image services, authentication and the hypervisor layer itself). 
So, the "integrated" tests in Nova are really not testing integration 
with other projects, but rather integration of the subsystems and 
processes inside Nova.

I'd support a policy that true integration tests -- tests that test the 
interaction between multiple real OpenStack service endpoints -- be left 
entirely to Tempest. Functional tests that test interaction between 
internal daemons and processes to a project should go into 
/$project/tests/functional/.

For Heat, I believe tests that rely on faked-out other OpenStack 
services but stress the interaction between internal Heat 
daemons/processes should be in /heat/tests/functional/ and any tests the 
rely on working, real OpenStack service endpoints should be in Tempest.

Best,
-jay