[Openstack] Integration tests

Gabe Westmaas gabe.westmaas at RACKSPACE.COM
Mon Sep 12 20:16:43 UTC 2011


On 9/12/11 2:54 PM, "Tim Simpson" <tim.simpson at rackspace.com> wrote:

>It would be advantageous for the tests to only use one client, whatever
>it turns out to be. I think it would be most practical to use a bleeding
>edge version of Nova client and add any features necessary to it in its
>own repo directly or by somehow mixing it in with the test code.

My biggest concern in using novaclient exclusively is masking errors in
both the API and novaclient.  If this is resolved another way, then I'm
ok.  There are a few different ways to do this, including running through
a separate python client that hits the API directly (as kong and
stacktester do, I believe), or using language bindings from other
languages as a part of the tests to make sure everyone is in sync.

Of course it is important that we maintain novaclient as a part of the end
to end tests as well.

Gabe

>
>As for Dtest- I guess great minds think alike? It looks like both
>projects started at the same time (Proboscis began back in February in an
>internal Rackspace repo) so its unfortunate we're only just now
>discovering each other's work. Dtest looks very interesting and the
>ability to run tests in parallel is something I've been thought about
>doing in Proboscis, but unsure of how to proceed since Proboscis orders
>things before sending the list to Nose (for the purposes of backwards
>compatibility). It may be possible however for Proboscis to feed Dtest's
>execution routines.
>
>At first glance the ordering functionality in dtest is similar to
>Proboscis but after reviewing some of the code in backfire I believe
>there are features of Proboscis not present there. Proboscis in general
>is TestNG in Python, and anytime I've broken away from how TestNG handles
>these things (usually on accident) I've found it to be a mistake-TestNG
>really did think out how to do this pretty well.
>
>We should talk sometime, I think each project could gain a lot from it.
>
>________________________________________
>From: Matt Dietz
>Sent: Monday, September 12, 2011 12:22 PM
>To: Tim Simpson; Jay Pipes
>Cc: openstack at lists.launchpad.net
>Subject: Re: [Openstack] Integration tests
>
>Regarding novaclient, I was debating this myself last night. Our backfire
>suite currently uses it. However, I can see scenarios where nova has
>advanced but novaclient has yet to, and because all three projects are
>independently developed, we're stuck temporarily. I can see utility in
>tests specifically for Novaclient, and ones that use a built in client (as
>the suite that Soren provided currently does.)
>
>Also, Backfire uses a module Kevin Mitchell wrote named Dtest that
>actually provides a lot of that same functionality you mention, Tim. It
>also gives you the ability to spawn the tests individually in greenthreads
>so they execute in parallel. We got push back when we initially tried to
>merge our tests into Nova, partially because it uses a 3rd party library
>for executing the tests. We can try merging these things together, if
>that's what everyone wants.
>
>On 9/12/11 11:39 AM, "Tim Simpson" <tim.simpson at rackspace.com> wrote:
>
>>I'm with the Reddwarf (Database as a Service) team at Rackspace. We
>>employ a large set of integration / functional tests which run in a
>>Vagrant controlled VM.
>>
>>Our tests use python-novaclient as well as a similar client for Reddwarf
>>we've written ourselves. The motivation is to eat our own dog food- if
>>the novaclient is the official rich client for Python, why shouldn't the
>>test make use of it?
>>
>>We also use a library called Proboscis so we can order our tests without
>>prefixing numbers to them, automatically skip related tests when
>>something in a dependency fails, and avoid global variables even as we
>>pass state between related tests. I think the openstack-integration-tests
>>would definitely benefit from it.
>>
>>I'd love to have a conversation about getting the traits outlined above
>>adopted into this standardized testing solution. :)
>>
>>Output of our tests running: http://pastie.org/2521835
>>Proboscis documentation: http://packages.python.org/proboscis/
>>________________________________________
>>From: openstack-bounces+tim.simpson=rackspace.com at lists.launchpad.net
>>[openstack-bounces+tim.simpson=rackspace.com at lists.launchpad.net] on
>>behalf of Jay Pipes [jaypipes at gmail.com]
>>Sent: Monday, September 12, 2011 8:20 AM
>>To: Matt Dietz
>>Cc: openstack at lists.launchpad.net
>>Subject: Re: [Openstack] Integration tests
>>
>>Wow, another integration test framework emerges out of the woodwork :)
>>
>>Looking forward to getting all of these into a single place... and
>>clearing up the lack of communication between teams on this subject!
>>There's Kong, Stacktester, and Backfire. Any others out there we
>>should know about?
>>
>>Cheers,
>>jay
>>
>>On Sun, Sep 11, 2011 at 9:09 PM, Matt Dietz <matt.dietz at rackspace.com>
>>wrote:
>>> Suite looks great, Soren!
>>> Wanted to mention that we actually developed our own suite of tests,
>>>which
>>> you can find at https://github.com/ohthree/backfire I'm planning to
>>> reconcile the difference so we can stop the independent efforts, but
>>>it's
>>> going to take time. Something else I'd like to see in the suite, that
>>>we
>>> currently have in backfire via a custom test module, is the ability to
>>> parallelize the tests for execution.
>>> From: Soren Hansen <soren at linux2go.dk>
>>> Date: Sat, 10 Sep 2011 00:21:10 +0200
>>> To: <openstack at lists.launchpad.net>
>>> Subject: Re: [Openstack] Integration tests
>>>
>>> That link shouldn't have included the +bug bit. Copy/paste fail. :(
>>>
>>> Sent from my phone. Pardon my brevity.
>>>
>>> _______________________________________________ Mailing list:
>>> https://launchpad.net/~openstack Post to :
>>>openstack at lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~openstack More help :
>>> https://help.launchpad.net/ListHelp This email may include confidential
>>> information. If you received it in error, please delete it.
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~openstack
>>> Post to     : openstack at lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~openstack
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>>_______________________________________________
>>Mailing list: https://launchpad.net/~openstack
>>Post to     : openstack at lists.launchpad.net
>>Unsubscribe : https://launchpad.net/~openstack
>>More help   : https://help.launchpad.net/ListHelp
>
>This email may include confidential information. If you received it in
>error, please delete it.
>
>
>_______________________________________________
>Mailing list: https://launchpad.net/~openstack
>Post to     : openstack at lists.launchpad.net
>Unsubscribe : https://launchpad.net/~openstack
>More help   : https://help.launchpad.net/ListHelp

This email may include confidential information. If you received it in error, please delete it.





More information about the Openstack mailing list