Open Stack

It is a big surprise.

I tough we choose a path outside from the tempest/ directory because that moment we didn't wanted to add it to the full tempest-devstack gate. 
We discussed about some kind of incubation "workflow".

When does it changed ? What is the current agreement about the incubation ?

When will the CLI tests verify the output ?  

----- Original Message -----
From: "Joe Gordon" <jogo at cloudscaling.com>
To: "All Things QA." <openstack-qa at lists.openstack.org>
Sent: Thursday, February 21, 2013 7:47:35 PM
Subject: Re: [openstack-qa] New Client Test Suite


Hi All, 


I am pleased to say that the CLI tests are now gating ( https://review.openstack.org/#/c/22096/ ). They run at the end of the standard tempest test (example: http://logs.openstack.org/22546/5/check/gate-tempest-devstack-vm-full/4878/console.html.gz ). 


The current set of tests are very basic and need a lot of work. But now that the CLI tests are self-gating it should be easy for anyone to contribute to them. Currently we have: 
* uses tempest.conf 
* in unittest framework 
* admin user only 
* python-novaclient only 
* read-only tests 


But we would like to have: 
* Tests for all python-*clients 
* Tests for all *-manage binaries 
* Tests for both admin and regular users 
* Read/Write tests (spin up a VM with client do something etc) 






Daryl, as for your WIP patch, I will take a look at it hopefully there are some things we can use from it. 


best, 
Joe Gordon 


On Thu, Feb 21, 2013 at 8:18 AM, Daryl Walleck < daryl.walleck at rackspace.com > wrote: 


Ahh, I didn't realize we an agreed upon solution for command line testing already. If there's nothing of use that can be pulled from what I submitted, I'll withdraw the branch. 

Daryl 
________________________________________ 
From: Jay Pipes [ jaypipes at gmail.com ] 
Sent: Thursday, February 21, 2013 9:44 AM 
To: openstack-qa at lists.openstack.org 


Subject: Re: [openstack-qa] New Client Test Suite 

On 02/21/2013 10:24 AM, Daryl Walleck wrote: 
> It took me a bit longer than I hoped, but I but a basic outline of some of the command line tests we have out in a WIP branch. If there's enough interest, I can get this merged into Tempest properly. 
> 
> https://review.openstack.org/#/c/22552/ 

Hi Daryl! 

We now have a CLI test module in Tempest: 

https://github.com/openstack/tempest/tree/master/cli 

If we bring in your CLI tests, I think I'd prefer to: 

a) Keep the code style consistent with the rest of Tempest 
b) Use the simple framework that Joe put together for the cli tests 
currently in Tempest 

Best, 
-jay 

> Daryl 
> ________________________________________ 
> From: Sam Danes [ sam.danes at RACKSPACE.COM ] 
> Sent: Thursday, February 14, 2013 11:15 AM 
> To: All Things QA. 
> Subject: Re: [openstack-qa] New Client Test Suite 
> 
> I've given Daryl a pointer to the prototype code we wrote. He'll work on getting it out to the community. 
> 
> -----Original Message----- 
> From: Attila Fazekas [mailto: afazekas at redhat.com ] 
> Sent: Thursday, February 14, 2013 7:33 AM 
> To: All Things QA. 
> Subject: Re: [openstack-qa] New Client Test Suite 
> 
> Very good idea! 
> 
> I think it can fit into tempest repository, but in a different directory. 
> 
> I would extend a scope too. 
> 
> At the first Round: 
> - *mange command tests. 
> - remote execution (with ssh connection pooling) The whitebox test type (not the existing test case), partially should be converted to utilities, partially the functionality should move to the new folder. 
> https://github.com/openstack/tempest/blob/master/tempest/whitebox.py#L138 
> 
> 
> ----- Original Message ----- 
>> From: "Daryl Walleck" < daryl.walleck at RACKSPACE.COM > 
>> To: "All Things QA." < openstack-qa at lists.openstack.org > 
>> Sent: Thursday, February 14, 2013 3:33:20 AM 
>> Subject: Re: [openstack-qa] New Client Test Suite 
>> 
>> 
>> 
>> 
>> 
>> We've bumped into some Novaclient only issues before in the past as 
>> well. Sam and some of his team did prototype some tests that execute 
>> using Novaclient. If there's any interest I can see about getting it 
>> into a WIP branch to get eyes on it. 
>> 
>> 
>> 
>> Daryl 
>> 
>> 
>> From: Joe Gordon [ jogo at cloudscaling.com ] 
>> Sent: Tuesday, February 12, 2013 4:39 PM 
>> To: openstack-qa at lists.openstack.org 
>> Subject: [openstack-qa] New Client Test Suite 
>> 
>> 
>> 
>> 
>> Hi All, 
>> 
>> We currently have thorough API tests (Tempest) that do an excellent 
>> job of testing nova, glance, keystone, etc. But to date we don't have 
>> anything that tests the assorted python clients. 
>> 
>> 
>> Extensive python client tests, which would consist of mostly shell 
>> commands, would prevent many bugs from arising such as: 
>> https://bugs.launchpad.net/python-novaclient/+bug/1122958 . 
>> 
>> 
>> The question is where should theses tests live? 
>> 
>> 
>> * Devstack/smoketests - these are just smoketests and are not designed 
>> to be exhaustive 
>> * Tempest - currently explicitly doesn't use the python clients, but 
>> we can add a separate folder that tests the clients 
>> * In respective client repos - easier to require new tests when adding 
>> to client, but will result in divergent test suites 
>> * Separate Repo - perhaps this is premature? 
>> 
>> 
>> I propose adding a separate folder in tempest to run the python client 
>> tests, and taking it from there. 
>> 
>> 
>> 
>> 
>> best, 
>> Joe Gordon 
>> _______________________________________________ 
>> openstack-qa mailing list 
>> openstack-qa at lists.openstack.org 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa 
>> 
> 
> _______________________________________________ 
> openstack-qa mailing list 
> openstack-qa at lists.openstack.org 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa 
> 
> _______________________________________________ 
> openstack-qa mailing list 
> openstack-qa at lists.openstack.org 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa 
> 
> _______________________________________________ 
> openstack-qa mailing list 
> openstack-qa at lists.openstack.org 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa 
> 

_______________________________________________ 
openstack-qa mailing list 
openstack-qa at lists.openstack.org 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa 

_______________________________________________ 
openstack-qa mailing list 
openstack-qa at lists.openstack.org 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa 


_______________________________________________
openstack-qa mailing list
openstack-qa at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa