Open Stack

On 21/02/13 21:00, David Kranz wrote:
> Joe, that's great. This is part of the wonderful explosion of tests 
> being contributed to Tempest. I am concerned about the performance of 
> the gate. Do we really want to run a complete test of novaclient every 
> time some one checks something into swift? Or any project other than 
> nova and novaclient? I am not trying to single out this change but the 
> current situation is that a test either gates all projects, or none. I 
> don't see how this is sustainable. I have submitted a summit session 
> about this http://summit.openstack.org/cfp/details/23.
>
>  -David

Depends on the client and the project, I guess.
AFAIK (client:users):

keystoneclient:
   - Everything, servers and clients

novaclient:
   - Horizon

cinderclient:
   - Horizon
   - Nova

quantumclient:
   - Horizon
   - Nova

glanceclient:
   - Horizon
   - Nova

Y.

>
> On 2/21/2013 1:47 PM, Joe Gordon wrote:
>> Hi All,
>>
>> I am pleased to say that the CLI tests are now gating 
>> (https://review.openstack.org/#/c/22096/). They run at the end of the 
>> standard tempest test (example: 
>> http://logs.openstack.org/22546/5/check/gate-tempest-devstack-vm-full/4878/console.html.gz). 
>>
>>
>> The current set of tests are very basic and need a lot of work.  But 
>> now that the CLI tests are self-gating it should be easy for anyone 
>> to contribute to them.  Currently we have:
>> * uses tempest.conf
>> * in unittest framework
>> * admin user only
>> * python-novaclient only
>> * read-only tests
>>
>> But we would like to have:
>> * Tests for all python-*clients
>> * Tests for all *-manage binaries
>> * Tests for both admin and regular users
>> * Read/Write tests (spin up a VM with client do something etc)
>>
>>
>> Daryl, as for your WIP patch, I will take a look at it hopefully 
>> there are some things we can use from it.
>>
>> best,
>> Joe Gordon
>>
>> On Thu, Feb 21, 2013 at 8:18 AM, Daryl Walleck 
>> <daryl.walleck at rackspace.com <mailto:daryl.walleck at rackspace.com>> wrote:
>>
>>     Ahh, I didn't realize we an agreed upon solution for command line
>>     testing already. If there's nothing of use that can be pulled
>>     from what I submitted, I'll withdraw the branch.
>>
>>     Daryl
>>     ________________________________________
>>     From: Jay Pipes [jaypipes at gmail.com <mailto:jaypipes at gmail.com>]
>>     Sent: Thursday, February 21, 2013 9:44 AM
>>     To: openstack-qa at lists.openstack.org
>>     <mailto:openstack-qa at lists.openstack.org>
>>     Subject: Re: [openstack-qa] New Client Test Suite
>>
>>     On 02/21/2013 10:24 AM, Daryl Walleck wrote:
>>     > It took me a bit longer than I hoped, but I but a basic outline
>>     of some of the command line tests we have out in a WIP branch. If
>>     there's enough interest, I can get this merged into Tempest properly.
>>     >
>>     > https://review.openstack.org/#/c/22552/
>>
>>     Hi Daryl!
>>
>>     We now have a CLI test module in Tempest:
>>
>>     https://github.com/openstack/tempest/tree/master/cli
>>
>>     If we bring in your CLI tests, I think I'd prefer to:
>>
>>     a) Keep the code style consistent with the rest of Tempest
>>     b) Use the simple framework that Joe put together for the cli tests
>>     currently in Tempest
>>
>>     Best,
>>     -jay
>>
>>     > Daryl
>>     > ________________________________________
>>     > From: Sam Danes [sam.danes at RACKSPACE.COM
>>     <mailto:sam.danes at RACKSPACE.COM>]
>>     > Sent: Thursday, February 14, 2013 11:15 AM
>>     > To: All Things QA.
>>     > Subject: Re: [openstack-qa] New Client Test Suite
>>     >
>>     > I've given Daryl a pointer to the prototype code we wrote.
>>     He'll work on getting it out to the community.
>>     >
>>     > -----Original Message-----
>>     > From: Attila Fazekas [mailto:afazekas at redhat.com
>>     <mailto:afazekas at redhat.com>]
>>     > Sent: Thursday, February 14, 2013 7:33 AM
>>     > To: All Things QA.
>>     > Subject: Re: [openstack-qa] New Client Test Suite
>>     >
>>     > Very good idea!
>>     >
>>     > I think it can fit into tempest repository,  but in a different
>>     directory.
>>     >
>>     > I would extend a scope too.
>>     >
>>     > At the first Round:
>>     > - *mange command tests.
>>     > - remote execution (with ssh connection pooling) The whitebox
>>     test type (not the existing test case), partially should be
>>     converted to utilities, partially the functionality should move
>>     to the new folder.
>>     >
>>     https://github.com/openstack/tempest/blob/master/tempest/whitebox.py#L138
>>     >
>>     >
>>     > ----- Original Message -----
>>     >> From: "Daryl Walleck" <daryl.walleck at RACKSPACE.COM
>>     <mailto:daryl.walleck at RACKSPACE.COM>>
>>     >> To: "All Things QA." <openstack-qa at lists.openstack.org
>>     <mailto:openstack-qa at lists.openstack.org>>
>>     >> Sent: Thursday, February 14, 2013 3:33:20 AM
>>     >> Subject: Re: [openstack-qa] New Client Test Suite
>>     >>
>>     >>
>>     >>
>>     >>
>>     >>
>>     >> We've bumped into some Novaclient only issues before in the
>>     past as
>>     >> well. Sam and some of his team did prototype some tests that
>>     execute
>>     >> using Novaclient. If there's any interest I can see about
>>     getting it
>>     >> into a WIP branch to get eyes on it.
>>     >>
>>     >>
>>     >>
>>     >> Daryl
>>     >>
>>     >>
>>     >> From: Joe Gordon [jogo at cloudscaling.com
>>     <mailto:jogo at cloudscaling.com>]
>>     >> Sent: Tuesday, February 12, 2013 4:39 PM
>>     >> To: openstack-qa at lists.openstack.org
>>     <mailto:openstack-qa at lists.openstack.org>
>>     >> Subject: [openstack-qa] New Client Test Suite
>>     >>
>>     >>
>>     >>
>>     >>
>>     >> Hi All,
>>     >>
>>     >> We currently have thorough API tests (Tempest) that do an
>>     excellent
>>     >> job of testing nova, glance, keystone, etc. But to date we
>>     don't have
>>     >> anything that tests the assorted python clients.
>>     >>
>>     >>
>>     >> Extensive python client tests, which would consist of mostly shell
>>     >> commands, would prevent many bugs from arising such as:
>>     >> https://bugs.launchpad.net/python-novaclient/+bug/1122958 .
>>     >>
>>     >>
>>     >> The question is where should theses tests live?
>>     >>
>>     >>
>>     >> * Devstack/smoketests - these are just smoketests and are not
>>     designed
>>     >> to be exhaustive
>>     >> * Tempest - currently explicitly doesn't use the python
>>     clients, but
>>     >> we can add a separate folder that tests the clients
>>     >> * In respective client repos - easier to require new tests
>>     when adding
>>     >> to client, but will result in divergent test suites
>>     >> * Separate Repo - perhaps this is premature?
>>     >>
>>     >>
>>     >> I propose adding a separate folder in tempest to run the
>>     python client
>>     >> tests, and taking it from there.
>>     >>
>>     >>
>>     >>
>>     >>
>>     >> best,
>>     >> Joe Gordon
>>     >> _______________________________________________
>>     >> openstack-qa mailing list
>>     >> openstack-qa at lists.openstack.org
>>     <mailto:openstack-qa at lists.openstack.org>
>>     >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>>     >>
>>     >
>>     > _______________________________________________
>>     > openstack-qa mailing list
>>     > openstack-qa at lists.openstack.org
>>     <mailto:openstack-qa at lists.openstack.org>
>>     > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>>     >
>>     > _______________________________________________
>>     > openstack-qa mailing list
>>     > openstack-qa at lists.openstack.org
>>     <mailto:openstack-qa at lists.openstack.org>
>>     > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>>     >
>>     > _______________________________________________
>>     > openstack-qa mailing list
>>     > openstack-qa at lists.openstack.org
>>     <mailto:openstack-qa at lists.openstack.org>
>>     > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>>     >
>>
>>     _______________________________________________
>>     openstack-qa mailing list
>>     openstack-qa at lists.openstack.org
>>     <mailto:openstack-qa at lists.openstack.org>
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>>
>>     _______________________________________________
>>     openstack-qa mailing list
>>     openstack-qa at lists.openstack.org
>>     <mailto:openstack-qa at lists.openstack.org>
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>>
>>
>>
>>
>> _______________________________________________
>> openstack-qa mailing list
>> openstack-qa at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>
>
>
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-qa/attachments/20130221/bac0276f/attachment-0001.html>