Open Stack

Joe, that's great. This is part of the wonderful explosion of tests 
being contributed to Tempest. I am concerned about the performance of 
the gate. Do we really want to run a complete test of novaclient every 
time some one checks something into swift? Or any project other than 
nova and novaclient? I am not trying to single out this change but the 
current situation is that a test either gates all projects, or none. I 
don't see how this is sustainable. I have submitted a summit session 
about this http://summit.openstack.org/cfp/details/23.

  -David

On 2/21/2013 1:47 PM, Joe Gordon wrote:
> Hi All,
>
> I am pleased to say that the CLI tests are now gating 
> (https://review.openstack.org/#/c/22096/). They run at the end of the 
> standard tempest test (example: 
> http://logs.openstack.org/22546/5/check/gate-tempest-devstack-vm-full/4878/console.html.gz). 
>
>
> The current set of tests are very basic and need a lot of work.  But 
> now that the CLI tests are self-gating it should be easy for anyone to 
> contribute to them.  Currently we have:
> * uses tempest.conf
> * in unittest framework
> * admin user only
> * python-novaclient only
> * read-only tests
>
> But we would like to have:
> * Tests for all python-*clients
> * Tests for all *-manage binaries
> * Tests for both admin and regular users
> * Read/Write tests (spin up a VM with client do something etc)
>
>
> Daryl, as for your WIP patch, I will take a look at it hopefully there 
> are some things we can use from it.
>
> best,
> Joe Gordon
>
> On Thu, Feb 21, 2013 at 8:18 AM, Daryl Walleck 
> <daryl.walleck at rackspace.com <mailto:daryl.walleck at rackspace.com>> wrote:
>
>     Ahh, I didn't realize we an agreed upon solution for command line
>     testing already. If there's nothing of use that can be pulled from
>     what I submitted, I'll withdraw the branch.
>
>     Daryl
>     ________________________________________
>     From: Jay Pipes [jaypipes at gmail.com <mailto:jaypipes at gmail.com>]
>     Sent: Thursday, February 21, 2013 9:44 AM
>     To: openstack-qa at lists.openstack.org
>     <mailto:openstack-qa at lists.openstack.org>
>     Subject: Re: [openstack-qa] New Client Test Suite
>
>     On 02/21/2013 10:24 AM, Daryl Walleck wrote:
>     > It took me a bit longer than I hoped, but I but a basic outline
>     of some of the command line tests we have out in a WIP branch. If
>     there's enough interest, I can get this merged into Tempest properly.
>     >
>     > https://review.openstack.org/#/c/22552/
>
>     Hi Daryl!
>
>     We now have a CLI test module in Tempest:
>
>     https://github.com/openstack/tempest/tree/master/cli
>
>     If we bring in your CLI tests, I think I'd prefer to:
>
>     a) Keep the code style consistent with the rest of Tempest
>     b) Use the simple framework that Joe put together for the cli tests
>     currently in Tempest
>
>     Best,
>     -jay
>
>     > Daryl
>     > ________________________________________
>     > From: Sam Danes [sam.danes at RACKSPACE.COM
>     <mailto:sam.danes at RACKSPACE.COM>]
>     > Sent: Thursday, February 14, 2013 11:15 AM
>     > To: All Things QA.
>     > Subject: Re: [openstack-qa] New Client Test Suite
>     >
>     > I've given Daryl a pointer to the prototype code we wrote. He'll
>     work on getting it out to the community.
>     >
>     > -----Original Message-----
>     > From: Attila Fazekas [mailto:afazekas at redhat.com
>     <mailto:afazekas at redhat.com>]
>     > Sent: Thursday, February 14, 2013 7:33 AM
>     > To: All Things QA.
>     > Subject: Re: [openstack-qa] New Client Test Suite
>     >
>     > Very good idea!
>     >
>     > I think it can fit into tempest repository,  but in a different
>     directory.
>     >
>     > I would extend a scope too.
>     >
>     > At the first Round:
>     > - *mange command tests.
>     > - remote execution (with ssh connection pooling) The whitebox
>     test type (not the existing test case), partially should be
>     converted to utilities, partially the functionality should move to
>     the new folder.
>     >
>     https://github.com/openstack/tempest/blob/master/tempest/whitebox.py#L138
>     >
>     >
>     > ----- Original Message -----
>     >> From: "Daryl Walleck" <daryl.walleck at RACKSPACE.COM
>     <mailto:daryl.walleck at RACKSPACE.COM>>
>     >> To: "All Things QA." <openstack-qa at lists.openstack.org
>     <mailto:openstack-qa at lists.openstack.org>>
>     >> Sent: Thursday, February 14, 2013 3:33:20 AM
>     >> Subject: Re: [openstack-qa] New Client Test Suite
>     >>
>     >>
>     >>
>     >>
>     >>
>     >> We've bumped into some Novaclient only issues before in the past as
>     >> well. Sam and some of his team did prototype some tests that
>     execute
>     >> using Novaclient. If there's any interest I can see about
>     getting it
>     >> into a WIP branch to get eyes on it.
>     >>
>     >>
>     >>
>     >> Daryl
>     >>
>     >>
>     >> From: Joe Gordon [jogo at cloudscaling.com
>     <mailto:jogo at cloudscaling.com>]
>     >> Sent: Tuesday, February 12, 2013 4:39 PM
>     >> To: openstack-qa at lists.openstack.org
>     <mailto:openstack-qa at lists.openstack.org>
>     >> Subject: [openstack-qa] New Client Test Suite
>     >>
>     >>
>     >>
>     >>
>     >> Hi All,
>     >>
>     >> We currently have thorough API tests (Tempest) that do an excellent
>     >> job of testing nova, glance, keystone, etc. But to date we
>     don't have
>     >> anything that tests the assorted python clients.
>     >>
>     >>
>     >> Extensive python client tests, which would consist of mostly shell
>     >> commands, would prevent many bugs from arising such as:
>     >> https://bugs.launchpad.net/python-novaclient/+bug/1122958 .
>     >>
>     >>
>     >> The question is where should theses tests live?
>     >>
>     >>
>     >> * Devstack/smoketests - these are just smoketests and are not
>     designed
>     >> to be exhaustive
>     >> * Tempest - currently explicitly doesn't use the python
>     clients, but
>     >> we can add a separate folder that tests the clients
>     >> * In respective client repos - easier to require new tests when
>     adding
>     >> to client, but will result in divergent test suites
>     >> * Separate Repo - perhaps this is premature?
>     >>
>     >>
>     >> I propose adding a separate folder in tempest to run the python
>     client
>     >> tests, and taking it from there.
>     >>
>     >>
>     >>
>     >>
>     >> best,
>     >> Joe Gordon
>     >> _______________________________________________
>     >> openstack-qa mailing list
>     >> openstack-qa at lists.openstack.org
>     <mailto:openstack-qa at lists.openstack.org>
>     >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>     >>
>     >
>     > _______________________________________________
>     > openstack-qa mailing list
>     > openstack-qa at lists.openstack.org
>     <mailto:openstack-qa at lists.openstack.org>
>     > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>     >
>     > _______________________________________________
>     > openstack-qa mailing list
>     > openstack-qa at lists.openstack.org
>     <mailto:openstack-qa at lists.openstack.org>
>     > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>     >
>     > _______________________________________________
>     > openstack-qa mailing list
>     > openstack-qa at lists.openstack.org
>     <mailto:openstack-qa at lists.openstack.org>
>     > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>     >
>
>     _______________________________________________
>     openstack-qa mailing list
>     openstack-qa at lists.openstack.org
>     <mailto:openstack-qa at lists.openstack.org>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>
>     _______________________________________________
>     openstack-qa mailing list
>     openstack-qa at lists.openstack.org
>     <mailto:openstack-qa at lists.openstack.org>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>
>
>
>
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-qa/attachments/20130221/6b9c6c5f/attachment-0001.html>