Open Stack

Hi All,

I am pleased to say that the CLI tests are now gating (
https://review.openstack.org/#/c/22096/). They run at the end of the
standard tempest test (example:
http://logs.openstack.org/22546/5/check/gate-tempest-devstack-vm-full/4878/console.html.gz
).

The current set of tests are very basic and need a lot of work.  But now
that the CLI tests are self-gating it should be easy for anyone to
contribute to them.  Currently we have:
* uses tempest.conf
* in unittest framework
* admin user only
* python-novaclient only
* read-only tests

But we would like to have:
* Tests for all python-*clients
* Tests for all *-manage binaries
* Tests for both admin and regular users
* Read/Write tests (spin up a VM with client do something etc)


Daryl, as for your WIP patch, I will take a look at it hopefully there are
some things we can use from it.

best,
Joe Gordon

On Thu, Feb 21, 2013 at 8:18 AM, Daryl Walleck
<daryl.walleck at rackspace.com>wrote:

> Ahh, I didn't realize we an agreed upon solution for command line testing
> already. If there's nothing of use that can be pulled from what I
> submitted, I'll withdraw the branch.
>
> Daryl
> ________________________________________
> From: Jay Pipes [jaypipes at gmail.com]
> Sent: Thursday, February 21, 2013 9:44 AM
> To: openstack-qa at lists.openstack.org
> Subject: Re: [openstack-qa] New Client Test Suite
>
> On 02/21/2013 10:24 AM, Daryl Walleck wrote:
> > It took me a bit longer than I hoped, but I but a basic outline of some
> of the command line tests we have out in a WIP branch. If there's enough
> interest, I can get this merged into Tempest properly.
> >
> > https://review.openstack.org/#/c/22552/
>
> Hi Daryl!
>
> We now have a CLI test module in Tempest:
>
> https://github.com/openstack/tempest/tree/master/cli
>
> If we bring in your CLI tests, I think I'd prefer to:
>
> a) Keep the code style consistent with the rest of Tempest
> b) Use the simple framework that Joe put together for the cli tests
> currently in Tempest
>
> Best,
> -jay
>
> > Daryl
> > ________________________________________
> > From: Sam Danes [sam.danes at RACKSPACE.COM]
> > Sent: Thursday, February 14, 2013 11:15 AM
> > To: All Things QA.
> > Subject: Re: [openstack-qa] New Client Test Suite
> >
> > I've given Daryl a pointer to the prototype code we wrote. He'll work on
> getting it out to the community.
> >
> > -----Original Message-----
> > From: Attila Fazekas [mailto:afazekas at redhat.com]
> > Sent: Thursday, February 14, 2013 7:33 AM
> > To: All Things QA.
> > Subject: Re: [openstack-qa] New Client Test Suite
> >
> > Very good idea!
> >
> > I think it can fit into tempest repository,  but in a different
> directory.
> >
> > I would extend a scope too.
> >
> > At the first Round:
> > - *mange command tests.
> > - remote execution (with ssh connection pooling) The whitebox test type
> (not the existing test case), partially should be converted to utilities,
> partially the functionality should move to the new folder.
> >
> https://github.com/openstack/tempest/blob/master/tempest/whitebox.py#L138
> >
> >
> > ----- Original Message -----
> >> From: "Daryl Walleck" <daryl.walleck at RACKSPACE.COM>
> >> To: "All Things QA." <openstack-qa at lists.openstack.org>
> >> Sent: Thursday, February 14, 2013 3:33:20 AM
> >> Subject: Re: [openstack-qa] New Client Test Suite
> >>
> >>
> >>
> >>
> >>
> >> We've bumped into some Novaclient only issues before in the past as
> >> well. Sam and some of his team did prototype some tests that execute
> >> using Novaclient. If there's any interest I can see about getting it
> >> into a WIP branch to get eyes on it.
> >>
> >>
> >>
> >> Daryl
> >>
> >>
> >> From: Joe Gordon [jogo at cloudscaling.com]
> >> Sent: Tuesday, February 12, 2013 4:39 PM
> >> To: openstack-qa at lists.openstack.org
> >> Subject: [openstack-qa] New Client Test Suite
> >>
> >>
> >>
> >>
> >> Hi All,
> >>
> >> We currently have thorough API tests (Tempest) that do an excellent
> >> job of testing nova, glance, keystone, etc. But to date we don't have
> >> anything that tests the assorted python clients.
> >>
> >>
> >> Extensive python client tests, which would consist of mostly shell
> >> commands, would prevent many bugs from arising such as:
> >> https://bugs.launchpad.net/python-novaclient/+bug/1122958 .
> >>
> >>
> >> The question is where should theses tests live?
> >>
> >>
> >> * Devstack/smoketests - these are just smoketests and are not designed
> >> to be exhaustive
> >> * Tempest - currently explicitly doesn't use the python clients, but
> >> we can add a separate folder that tests the clients
> >> * In respective client repos - easier to require new tests when adding
> >> to client, but will result in divergent test suites
> >> * Separate Repo - perhaps this is premature?
> >>
> >>
> >> I propose adding a separate folder in tempest to run the python client
> >> tests, and taking it from there.
> >>
> >>
> >>
> >>
> >> best,
> >> Joe Gordon
> >> _______________________________________________
> >> openstack-qa mailing list
> >> openstack-qa at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
> >>
> >
> > _______________________________________________
> > openstack-qa mailing list
> > openstack-qa at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
> >
> > _______________________________________________
> > openstack-qa mailing list
> > openstack-qa at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
> >
> > _______________________________________________
> > openstack-qa mailing list
> > openstack-qa at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
> >
>
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-qa/attachments/20130221/80d633b8/attachment.html>