Open Stack

+1 - we need some discussion on this as services has grown.

From: David Kranz [mailto:david.kranz at qrclab.com]
Sent: Thursday, February 21, 2013 11:00 AM
To: openstack-qa at lists.openstack.org
Subject: Re: [openstack-qa] New Client Test Suite

Joe, that's great. This is part of the wonderful explosion of tests being contributed to Tempest. I am concerned about the performance of the gate. Do we really want to run a complete test of novaclient every time some one checks something into swift? Or any project other than nova and novaclient? I am not trying to single out this change but the current situation is that a test either gates all projects, or none. I don't see how this is sustainable. I have submitted a summit session about this http://summit.openstack.org/cfp/details/23.

 -David

On 2/21/2013 1:47 PM, Joe Gordon wrote:
Hi All,

I am pleased to say that the CLI tests are now gating (https://review.openstack.org/#/c/22096/). They run at the end of the standard tempest test (example: http://logs.openstack.org/22546/5/check/gate-tempest-devstack-vm-full/4878/console.html.gz).

The current set of tests are very basic and need a lot of work.  But now that the CLI tests are self-gating it should be easy for anyone to contribute to them.  Currently we have:
* uses tempest.conf
* in unittest framework
* admin user only
* python-novaclient only
* read-only tests

But we would like to have:
* Tests for all python-*clients
* Tests for all *-manage binaries
* Tests for both admin and regular users
* Read/Write tests (spin up a VM with client do something etc)


Daryl, as for your WIP patch, I will take a look at it hopefully there are some things we can use from it.

best,
Joe Gordon

On Thu, Feb 21, 2013 at 8:18 AM, Daryl Walleck <daryl.walleck at rackspace.com<mailto:daryl.walleck at rackspace.com>> wrote:
Ahh, I didn't realize we an agreed upon solution for command line testing already. If there's nothing of use that can be pulled from what I submitted, I'll withdraw the branch.

Daryl
________________________________________
From: Jay Pipes [jaypipes at gmail.com<mailto:jaypipes at gmail.com>]
Sent: Thursday, February 21, 2013 9:44 AM
To: openstack-qa at lists.openstack.org<mailto:openstack-qa at lists.openstack.org>
Subject: Re: [openstack-qa] New Client Test Suite

On 02/21/2013 10:24 AM, Daryl Walleck wrote:
> It took me a bit longer than I hoped, but I but a basic outline of some of the command line tests we have out in a WIP branch. If there's enough interest, I can get this merged into Tempest properly.
>
> https://review.openstack.org/#/c/22552/

Hi Daryl!

We now have a CLI test module in Tempest:

https://github.com/openstack/tempest/tree/master/cli

If we bring in your CLI tests, I think I'd prefer to:

a) Keep the code style consistent with the rest of Tempest
b) Use the simple framework that Joe put together for the cli tests
currently in Tempest

Best,
-jay

> Daryl
> ________________________________________
> From: Sam Danes [sam.danes at RACKSPACE.COM<mailto:sam.danes at RACKSPACE.COM>]
> Sent: Thursday, February 14, 2013 11:15 AM
> To: All Things QA.
> Subject: Re: [openstack-qa] New Client Test Suite
>
> I've given Daryl a pointer to the prototype code we wrote. He'll work on getting it out to the community.
>
> -----Original Message-----
> From: Attila Fazekas [mailto:afazekas at redhat.com<mailto:afazekas at redhat.com>]
> Sent: Thursday, February 14, 2013 7:33 AM
> To: All Things QA.
> Subject: Re: [openstack-qa] New Client Test Suite
>
> Very good idea!
>
> I think it can fit into tempest repository,  but in a different directory.
>
> I would extend a scope too.
>
> At the first Round:
> - *mange command tests.
> - remote execution (with ssh connection pooling) The whitebox test type (not the existing test case), partially should be converted to utilities, partially the functionality should move to the new folder.
> https://github.com/openstack/tempest/blob/master/tempest/whitebox.py#L138
>
>
> ----- Original Message -----
>> From: "Daryl Walleck" <daryl.walleck at RACKSPACE.COM<mailto:daryl.walleck at RACKSPACE.COM>>
>> To: "All Things QA." <openstack-qa at lists.openstack.org<mailto:openstack-qa at lists.openstack.org>>
>> Sent: Thursday, February 14, 2013 3:33:20 AM
>> Subject: Re: [openstack-qa] New Client Test Suite
>>
>>
>>
>>
>>
>> We've bumped into some Novaclient only issues before in the past as
>> well. Sam and some of his team did prototype some tests that execute
>> using Novaclient. If there's any interest I can see about getting it
>> into a WIP branch to get eyes on it.
>>
>>
>>
>> Daryl
>>
>>
>> From: Joe Gordon [jogo at cloudscaling.com<mailto:jogo at cloudscaling.com>]
>> Sent: Tuesday, February 12, 2013 4:39 PM
>> To: openstack-qa at lists.openstack.org<mailto:openstack-qa at lists.openstack.org>
>> Subject: [openstack-qa] New Client Test Suite
>>
>>
>>
>>
>> Hi All,
>>
>> We currently have thorough API tests (Tempest) that do an excellent
>> job of testing nova, glance, keystone, etc. But to date we don't have
>> anything that tests the assorted python clients.
>>
>>
>> Extensive python client tests, which would consist of mostly shell
>> commands, would prevent many bugs from arising such as:
>> https://bugs.launchpad.net/python-novaclient/+bug/1122958 .
>>
>>
>> The question is where should theses tests live?
>>
>>
>> * Devstack/smoketests - these are just smoketests and are not designed
>> to be exhaustive
>> * Tempest - currently explicitly doesn't use the python clients, but
>> we can add a separate folder that tests the clients
>> * In respective client repos - easier to require new tests when adding
>> to client, but will result in divergent test suites
>> * Separate Repo - perhaps this is premature?
>>
>>
>> I propose adding a separate folder in tempest to run the python client
>> tests, and taking it from there.
>>
>>
>>
>>
>> best,
>> Joe Gordon
>> _______________________________________________
>> openstack-qa mailing list
>> openstack-qa at lists.openstack.org<mailto:openstack-qa at lists.openstack.org>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>>
>
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org<mailto:openstack-qa at lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org<mailto:openstack-qa at lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org<mailto:openstack-qa at lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>

_______________________________________________
openstack-qa mailing list
openstack-qa at lists.openstack.org<mailto:openstack-qa at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa

_______________________________________________
openstack-qa mailing list
openstack-qa at lists.openstack.org<mailto:openstack-qa at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa





_______________________________________________

openstack-qa mailing list

openstack-qa at lists.openstack.org<mailto:openstack-qa at lists.openstack.org>

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-qa/attachments/20130221/5c965bdb/attachment-0001.html>