Open Stack

On 26/08/14 18:59, Clint Byrum wrote:
> Excerpts from Steve Baker's message of 2014-08-26 14:25:46 -0700:
>> On 27/08/14 03:18, David Kranz wrote:
>>> On 08/26/2014 10:14 AM, Zane Bitter wrote:
>>>> Steve Baker has started the process of moving Heat tests out of the
>>>> Tempest repository and into the Heat repository, and we're looking
>>>> for some guidance on how they should be packaged in a consistent way.
>>>> Apparently there are a few projects already packaging functional
>>>> tests in the package <projectname>.tests.functional (alongside
>>>> <projectname>.tests.unit for the unit tests).
>>>>
>>>> That strikes me as odd in our context, because while the unit tests
>>>> run against the code in the package in which they are embedded, the
>>>> functional tests run against some entirely different code - whatever
>>>> OpenStack cloud you give it the auth URL and credentials for. So
>>>> these tests run from the outside, just like their ancestors in
>>>> Tempest do.
>>>>
>>>> There's all kinds of potential confusion here for users and
>>>> packagers. None of it is fatal and all of it can be worked around,
>>>> but if we refrain from doing the thing that makes zero conceptual
>>>> sense then there will be no problem to work around :)
>>> Thanks, Zane. The point of moving functional tests to projects is to
>>> be able to run more of them
>>> in gate jobs for those projects, and allow tempest to survive being
>>> stretched-to-breaking horizontally as we scale to more projects. At
>>> the same time, there are benefits to the
>>> tempest-as-all-in-one-functional-and-integration-suite that we should
>>> try not to lose:
>>>
>>> 1. Strong integration testing without thinking too hard about the
>>> actual dependencies
>>> 2. Protection from mistaken or unwise api changes (tempest two-step
>>> required)
>>> 3. Exportability as a complete blackbox functional test suite that can
>>> be used by Rally, RefStack, deployment validation, etc.
>>>
>>> I think (1) may be the most challenging because tests that are moved
>>> out of tempest might be testing some integration that is not being
>>> covered
>>> by a scenario. We will need to make sure that tempest actually has a
>>> complete enough set of tests to validate integration. Even if this is
>>> all implemented in a way where tempest can see in-project tests as
>>> "plugins", there will still not be time to run them all as part of
>>> tempest on every commit to every project, so a selection will have to
>>> be made.
>>>
>>> (2) is quite difficult. In Atlanta we talked about taking a copy of
>>> functional tests into tempest for stable apis. I don't know how
>>> workable that is but don't see any other real options except vigilance
>>> in reviews of patches that change functional tests.
>>>
>>> (3) is what Zane was addressing. The in-project functional tests need
>>> to be written in a way that they can, at least in some configuration,
>>> run against a real cloud.
>>>
>>>
>>>>
>>>> I suspect from reading the previous thread about "In-tree functional
>>>> test vision" that we may actually be dealing with three categories of
>>>> test here rather than two:
>>>>
>>>> * Unit tests that run against the package they are embedded in
>>>> * Functional tests that run against the package they are embedded in
>>>> * Integration tests that run against a specified cloud
>>>>
>>>> i.e. the tests we are now trying to add to Heat might be
>>>> qualitatively different from the <projectname>.tests.functional
>>>> suites that already exist in a few projects. Perhaps someone from
>>>> Neutron and/or Swift can confirm?
>>> That seems right, except that I would call the third "functional
>>> tests" and not "integration tests", because the purpose is not really
>>> integration but deep testing of a particular service. Tempest would
>>> continue to focus on integration testing. Is there some controversy
>>> about that?
>>> The second category could include whitebox tests.
>>>
>>> I don't know about swift, but in neutron the intent was to have these
>>> tests be configurable to run against a real cloud, or not. Maru Newby
>>> would have details.
>>>>
>>>> I'd like to propose that tests of the third type get their own
>>>> top-level package with a name of the form
>>>> <projectname>-integrationtests (second choice: <projectname>-tempest
>>>> on the principle that they're essentially plugins for Tempest). How
>>>> would people feel about standardising that across OpenStack?
>>> +1 But I would not call it "integrationtests" for the reason given above.
>>>
>> Because all heat does is interact with other services, what we call
>> functional tests are actually integration tests. Sure, we could mock at
>> the REST API level, but integration coverage is what we need most. This
>
> I'd call that "faking", not mocking, but both could apply.
>
>> lets us verify things like:
>> - how heat handles races in other services leading to resources going
>> into ERROR
>
> A fake that predictably fails (and thus tests failure handling) will
> result in better coverage than a real service that only fails when that
> real service is broken. What's frustrating is that _both_ are needed to
> catch bugs.

Yeah, we discussed this in the project meeting yesterday[1] and 
basically came to the conclusion that we'll ultimately need both 
functional tests with fake services and integration tests with real 
services.

Also, the latter is uncharted territory, so we should just implement it 
in whatever way seems to make the most sense and we can decide later 
based on the experience whether it really did or not.

cheers,
Zane.

[1] 
http://eavesdrop.openstack.org/meetings/project/2014/project.2014-08-26-21.01.log.html

>> - connectivity and interaction between heat and agents on orchestrated
>> servers
>>
>
> That is definitely a real integration problem, as it also tests OS's,
> cloud-init, ec2 metadata, etc. etc.
>
> I'd like to see fake versions of each API service available (think nova
> fake virt, fake cinder, etc) and simple ways to spin them up with fakes.
>
> Of course full integration tests need more realistic simulation.
> However, being able to run a set of tests that do actually exercise the
> interactions between the real services should lead to more developers
> running them in a tight loop and less gate churn.
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>