Open Stack

On Thu, Feb 21, 2013 at 11:00 AM, David Kranz <david.kranz at qrclab.com>wrote:

>  Joe, that's great. This is part of the wonderful explosion of tests
> being contributed to Tempest. I am concerned about the performance of the
> gate. Do we really want to run a complete test of novaclient every time
> some one checks something into swift? Or any project other than nova and
> novaclient? I am not trying to single out this change but the current
> situation is that a test either gates all projects, or none. I don't see
> how this is sustainable. I have submitted a summit session about this
> http://summit.openstack.org/cfp/details/23.
>

While I agree that this warrants further discussion.  The current version
of the CLI tests take only a few seconds, and as they expand (and start
using testr) I don't expect the tests to make much longer than a minute or
so, which I think is a reasonable amount of time to increase the gate by.

best,
Joe


>
>
>  -David
>
>
> On 2/21/2013 1:47 PM, Joe Gordon wrote:
>
> Hi All,
>
>  I am pleased to say that the CLI tests are now gating (
> https://review.openstack.org/#/c/22096/). They run at the end of the
> standard tempest test (example:
> http://logs.openstack.org/22546/5/check/gate-tempest-devstack-vm-full/4878/console.html.gz).
>
>
>  The current set of tests are very basic and need a lot of work.  But now
> that the CLI tests are self-gating it should be easy for anyone to
> contribute to them.  Currently we have:
> * uses tempest.conf
> * in unittest framework
> * admin user only
> * python-novaclient only
> * read-only tests
>
>  But we would like to have:
> * Tests for all python-*clients
>  * Tests for all *-manage binaries
> * Tests for both admin and regular users
> * Read/Write tests (spin up a VM with client do something etc)
>
>
>  Daryl, as for your WIP patch, I will take a look at it hopefully there
> are some things we can use from it.
>
>  best,
> Joe Gordon
>
> On Thu, Feb 21, 2013 at 8:18 AM, Daryl Walleck <
> daryl.walleck at rackspace.com> wrote:
>
>> Ahh, I didn't realize we an agreed upon solution for command line testing
>> already. If there's nothing of use that can be pulled from what I
>> submitted, I'll withdraw the branch.
>>
>> Daryl
>> ________________________________________
>> From: Jay Pipes [jaypipes at gmail.com]
>> Sent: Thursday, February 21, 2013 9:44 AM
>> To: openstack-qa at lists.openstack.org
>>  Subject: Re: [openstack-qa] New Client Test Suite
>>
>> On 02/21/2013 10:24 AM, Daryl Walleck wrote:
>> > It took me a bit longer than I hoped, but I but a basic outline of some
>> of the command line tests we have out in a WIP branch. If there's enough
>> interest, I can get this merged into Tempest properly.
>> >
>> > https://review.openstack.org/#/c/22552/
>>
>> Hi Daryl!
>>
>> We now have a CLI test module in Tempest:
>>
>> https://github.com/openstack/tempest/tree/master/cli
>>
>> If we bring in your CLI tests, I think I'd prefer to:
>>
>> a) Keep the code style consistent with the rest of Tempest
>> b) Use the simple framework that Joe put together for the cli tests
>> currently in Tempest
>>
>> Best,
>> -jay
>>
>> > Daryl
>> > ________________________________________
>> > From: Sam Danes [sam.danes at RACKSPACE.COM]
>> > Sent: Thursday, February 14, 2013 11:15 AM
>> > To: All Things QA.
>> > Subject: Re: [openstack-qa] New Client Test Suite
>> >
>> > I've given Daryl a pointer to the prototype code we wrote. He'll work
>> on getting it out to the community.
>> >
>> > -----Original Message-----
>> > From: Attila Fazekas [mailto:afazekas at redhat.com]
>> > Sent: Thursday, February 14, 2013 7:33 AM
>> > To: All Things QA.
>> > Subject: Re: [openstack-qa] New Client Test Suite
>> >
>> > Very good idea!
>> >
>> > I think it can fit into tempest repository,  but in a different
>> directory.
>> >
>> > I would extend a scope too.
>> >
>> > At the first Round:
>> > - *mange command tests.
>> > - remote execution (with ssh connection pooling) The whitebox test type
>> (not the existing test case), partially should be converted to utilities,
>> partially the functionality should move to the new folder.
>> >
>> https://github.com/openstack/tempest/blob/master/tempest/whitebox.py#L138
>> >
>> >
>> > ----- Original Message -----
>> >> From: "Daryl Walleck" <daryl.walleck at RACKSPACE.COM>
>> >> To: "All Things QA." <openstack-qa at lists.openstack.org>
>> >> Sent: Thursday, February 14, 2013 3:33:20 AM
>> >> Subject: Re: [openstack-qa] New Client Test Suite
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> We've bumped into some Novaclient only issues before in the past as
>> >> well. Sam and some of his team did prototype some tests that execute
>> >> using Novaclient. If there's any interest I can see about getting it
>> >> into a WIP branch to get eyes on it.
>> >>
>> >>
>> >>
>> >> Daryl
>> >>
>> >>
>> >> From: Joe Gordon [jogo at cloudscaling.com]
>> >> Sent: Tuesday, February 12, 2013 4:39 PM
>> >> To: openstack-qa at lists.openstack.org
>> >> Subject: [openstack-qa] New Client Test Suite
>> >>
>> >>
>> >>
>> >>
>> >> Hi All,
>> >>
>> >> We currently have thorough API tests (Tempest) that do an excellent
>> >> job of testing nova, glance, keystone, etc. But to date we don't have
>> >> anything that tests the assorted python clients.
>> >>
>> >>
>> >> Extensive python client tests, which would consist of mostly shell
>> >> commands, would prevent many bugs from arising such as:
>> >> https://bugs.launchpad.net/python-novaclient/+bug/1122958 .
>> >>
>> >>
>> >> The question is where should theses tests live?
>> >>
>> >>
>> >> * Devstack/smoketests - these are just smoketests and are not designed
>> >> to be exhaustive
>> >> * Tempest - currently explicitly doesn't use the python clients, but
>> >> we can add a separate folder that tests the clients
>> >> * In respective client repos - easier to require new tests when adding
>> >> to client, but will result in divergent test suites
>> >> * Separate Repo - perhaps this is premature?
>> >>
>> >>
>> >> I propose adding a separate folder in tempest to run the python client
>> >> tests, and taking it from there.
>> >>
>> >>
>> >>
>> >>
>> >> best,
>> >> Joe Gordon
>> >> _______________________________________________
>> >> openstack-qa mailing list
>> >> openstack-qa at lists.openstack.org
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>> >>
>> >
>> > _______________________________________________
>> > openstack-qa mailing list
>> > openstack-qa at lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>> >
>> > _______________________________________________
>> > openstack-qa mailing list
>> > openstack-qa at lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>> >
>> > _______________________________________________
>> > openstack-qa mailing list
>> > openstack-qa at lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>> >
>>
>> _______________________________________________
>> openstack-qa mailing list
>> openstack-qa at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>>
>> _______________________________________________
>> openstack-qa mailing list
>> openstack-qa at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>>
>
>
>
> _______________________________________________
> openstack-qa mailing listopenstack-qa at lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>
>
>
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-qa/attachments/20130221/c9ce6672/attachment.html>