Open Stack

On Fri, Sep 5, 2014 at 2:40 AM, Jay Pipes <jaypipes at gmail.com> wrote:

> On 09/04/2014 11:32 AM, Steven Hardy wrote:
>
>> On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote:
>>
>>> On 08/29/2014 05:15 PM, Zane Bitter wrote:
>>>
>>>> On 29/08/14 14:27, Jay Pipes wrote:
>>>>
>>>>> On 08/26/2014 10:14 AM, Zane Bitter wrote:
>>>>>
>>>>>> Steve Baker has started the process of moving Heat tests out of the
>>>>>> Tempest repository and into the Heat repository, and we're looking for
>>>>>> some guidance on how they should be packaged in a consistent way.
>>>>>> Apparently there are a few projects already packaging functional tests
>>>>>> in the package <projectname>.tests.functional (alongside
>>>>>> <projectname>.tests.unit for the unit tests).
>>>>>>
>>>>>> That strikes me as odd in our context, because while the unit tests
>>>>>> run
>>>>>> against the code in the package in which they are embedded, the
>>>>>> functional tests run against some entirely different code - whatever
>>>>>> OpenStack cloud you give it the auth URL and credentials for. So these
>>>>>> tests run from the outside, just like their ancestors in Tempest do.
>>>>>>
>>>>>> There's all kinds of potential confusion here for users and packagers.
>>>>>> None of it is fatal and all of it can be worked around, but if we
>>>>>> refrain from doing the thing that makes zero conceptual sense then
>>>>>> there
>>>>>> will be no problem to work around :)
>>>>>>
>>>>>> I suspect from reading the previous thread about "In-tree functional
>>>>>> test vision" that we may actually be dealing with three categories of
>>>>>> test here rather than two:
>>>>>>
>>>>>> * Unit tests that run against the package they are embedded in
>>>>>> * Functional tests that run against the package they are embedded in
>>>>>> * Integration tests that run against a specified cloud
>>>>>>
>>>>>> i.e. the tests we are now trying to add to Heat might be qualitatively
>>>>>> different from the <projectname>.tests.functional suites that already
>>>>>> exist in a few projects. Perhaps someone from Neutron and/or Swift can
>>>>>> confirm?
>>>>>>
>>>>>> I'd like to propose that tests of the third type get their own
>>>>>> top-level
>>>>>> package with a name of the form <projectname>-integrationtests (second
>>>>>> choice: <projectname>-tempest on the principle that they're
>>>>>> essentially
>>>>>> plugins for Tempest). How would people feel about standardising that
>>>>>> across OpenStack?
>>>>>>
>>>>>
>>>>> By its nature, Heat is one of the only projects that would have
>>>>> integration tests of this nature. For Nova, there are some "functional"
>>>>> tests in nova/tests/integrated/ (yeah, badly named, I know) that are
>>>>> tests of the REST API endpoints and running service daemons (the things
>>>>> that are RPC endpoints), with a bunch of stuff faked out (like RPC
>>>>> comms, image services, authentication and the hypervisor layer itself).
>>>>> So, the "integrated" tests in Nova are really not testing integration
>>>>> with other projects, but rather integration of the subsystems and
>>>>> processes inside Nova.
>>>>>
>>>>> I'd support a policy that true integration tests -- tests that test the
>>>>> interaction between multiple real OpenStack service endpoints -- be
>>>>> left
>>>>> entirely to Tempest. Functional tests that test interaction between
>>>>> internal daemons and processes to a project should go into
>>>>> /$project/tests/functional/.
>>>>>
>>>>> For Heat, I believe tests that rely on faked-out other OpenStack
>>>>> services but stress the interaction between internal Heat
>>>>> daemons/processes should be in /heat/tests/functional/ and any tests
>>>>> the
>>>>> rely on working, real OpenStack service endpoints should be in Tempest.
>>>>>
>>>>
>>>> Well, the problem with that is that last time I checked there was
>>>> exactly one Heat scenario test in Tempest because tempest-core doesn't
>>>> have the bandwidth to merge all (any?) of the other ones folks
>>>> submitted.
>>>>
>>>> So we're moving them to openstack/heat for the pure practical reason
>>>> that it's the only way to get test coverage at all, rather than concerns
>>>> about overloading the gate or theories about the best venue for
>>>> cross-project integration testing.
>>>>
>>>
>>> Hmm, speaking of passive aggressivity...
>>>
>>> Where can I see a discussion of the Heat integration tests with Tempest
>>> QA
>>> folks? If you give me some background on what efforts have been made
>>> already
>>> and what is remaining to be reviewed/merged/worked on, then I can try to
>>> get
>>> some resources dedicated to helping here.
>>>
>>
>> We recieved some fairly strong criticism from sdague[1] earlier this year,
>> at which point we were  already actively working on improving test
>> coverage
>> by writing new tests for tempest.
>>
>> Since then, several folks, myself included, commited very significant
>> amounts of additional effort to writing more tests for tempest, with some
>> success.
>>
>> Ultimately the review latency and overhead involved in constantly rebasing
>> changes between infrequent reviews has resulted in slow progress and
>> significant frustration for those attempting to contribute new test cases.
>>
>> It's been clear for a while that tempest-core have significant bandwidth
>> issues, as well as not necessarily always having the specific domain
>> expertise to thoroughly review some tests related to project-specific
>> behavior or functionality.
>>
>> So it was with some relief that we saw the proposal[2] to move the burden
>> for reviewing project test-cases to the project teams, who will presumably
>> be more motivated to do the reviews, and have the knowledge of what needs
>> testing.
>>
>> [1] http://lists.openstack.org/pipermail/openstack-dev/2014-
>> March/029661.html
>> [2] http://lists.openstack.org/pipermail/openstack-dev/2014-
>> July/041057.html
>>
>>  I would greatly prefer just having a single source of integration
>>> testing in
>>> OpenStack, versus going back to the bad ol' days of everybody under the
>>> sun
>>> rewriting their own.
>>>
>>> Note that I'm not talking about functional testing here, just the
>>> integration testing...
>>>
>>
>> You may have to define the terms functional and integration here, as IMO
>> there's already significant confusion about what the target of e.g API and
>> scenario tests in tempest are.
>>
>
> Yes, I already did this in a previous (snipped) portion of this email
> thread. See here:
>
> http://lists.openstack.org/pipermail/openstack-dev/2014-August/044514.html
>
> yay for multi-month-crossing ML threads ;)
>
>
>  This is also further complicated by the fact that all heat functional
>> tests
>> also test integration of the various underlying services to some extent.
>>
>
> Yes, as acknowledged in above post -- but Heat is special in this regard,
> as mentioned in the post above. Functional tests for Heat are not the same
> as functional tests for Nova, Keystone, Glance, Swift, etc.
>
>
>  My opinion is that any tests remaining in tempest should focus on API
>> correctness, e.g to keep us honest in terms of backwards incomaptible
>> changes to the API surface.
>>
>
> OK, that's a perfectly reasonable suggestion. Would the Heat
> functional^Wintegration tests then only test the latest version of the APIs?
>
>
>  Then for all tests which aim to prove the functionality of the project,
>> e.g
>> my understanding of tempest scenario tests atm, we should allow project
>> teams to own them, and add to them as functionality develops over time.
>>
>
> Well, that's where things get interesting. Functional tests in Glance and
> Nova, for instance, are functional tests, not integration tests, because
> they specifically do NOT test things like authentication -- i.e. the
> interaction between Nova/Glance and Keystone. The Nova functional tests
> likewise do not test the interaction between Nova and Glance, nor Nova and
> the underlying hypervisor, as those layers are both faked out in the
> functional Nova tests.
>
> As I mentioned in the above post, though, Heat is different. In order for
> Heat to really have a worthwhile functional test, it would have to fake out
> every single OpenStack service, since Heat itself does little more than
> provide an


This might be possible:-O

Check out:
https://openstacksummitnovember2014paris.sched.org/event/78969d086807fb8f60b29142149d9163#.VAka3da5s1w
https://github.com/rackerlabs/mimic

It would be neat to have a "fakestack" that heat (and others) could use.

Then we run our functional tests against fakestack and the integration
tests run the same tests just with a real
devstack backing it.

-Angus


> orchestration REST API and template format that itself calls other
> OpenStack services. And it is for this reason that I believe functional
> tests for Heat belong in Tempest, since Tempest tests assume a working
> OpenStack environment already, and there's no reason to either fake out all
> the OpenStack services, nor any reason to develop a separate integration
> testing framework inside of Heat that does the same things that Tempest
> does.
>
> For Glance functional tests, they are testing the communication channels
> between the Glance API service and the Glance registry service. For Nova
> functional tests, they are testing the communication channels between the
> Nova API, Nova conductor, Nova scheduler and Nova compute services.
>
> Heat does not have these internal interfaces to test communications for,
> and this is why I don't think an in-Heat-tree functional test suite is
> worth bothering with versus the (yes, frustrating) process of adding
> tempest Heat tests.
>
> Best,
> -jay
>
>
>  Ultimately I don't think it really matters which repo those tests live in,
>> provided we can write them and get them running in the gate (catching
>> regressions, which otherwise keep slipping through) in a timely manner.
>>
>> Steve
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140905/d97a5b6c/attachment.html>