<br><br><div class="gmail_quote">On Mon, Feb 25, 2013 at 12:46 AM, Attila Fazekas <span dir="ltr"><<a href="mailto:afazekas@redhat.com" target="_blank">afazekas@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
It is a big surprise.<br>
<br>
I tough we choose a path outside from the tempest/ directory because that moment we didn't wanted to add it to the full tempest-devstack gate.<br>
We discussed about some kind of incubation "workflow".<br></blockquote><div><br></div><div>The code is in the tempest repo but outside of the tempest directory in that repo, making the CLI tests an addition existing project instead of a new project. Because of that, the incubation workflow was somewhat sidestepped.</div>
<div><br></div><div>I am open to rethinking what we have now. </div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
When does it changed ? What is the current agreement about the incubation ?<br>
<br>
When will the CLI tests verify the output ?<br></blockquote><div><br></div><div>As soon as someone writes the tests to verify output. Contributions are welcome.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><br>
----- Original Message -----<br>
From: "Joe Gordon" <<a href="mailto:jogo@cloudscaling.com" target="_blank">jogo@cloudscaling.com</a>><br>
To: "All Things QA." <<a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a>><br>
</div><div>Sent: Thursday, February 21, 2013 7:47:35 PM<br>
Subject: Re: [openstack-qa] New Client Test Suite<br>
<br>
<br>
</div>Hi All,<br>
<br>
<br>
I am pleased to say that the CLI tests are now gating ( <a href="https://review.openstack.org/#/c/22096/" target="_blank">https://review.openstack.org/#/c/22096/</a> ). They run at the end of the standard tempest test (example: <a href="http://logs.openstack.org/22546/5/check/gate-tempest-devstack-vm-full/4878/console.html.gz" target="_blank">http://logs.openstack.org/22546/5/check/gate-tempest-devstack-vm-full/4878/console.html.gz</a> ).<br>
<div><div><br>
<br>
The current set of tests are very basic and need a lot of work. But now that the CLI tests are self-gating it should be easy for anyone to contribute to them. Currently we have:<br>
* uses tempest.conf<br>
* in unittest framework<br>
* admin user only<br>
* python-novaclient only<br>
* read-only tests<br>
<br>
<br>
But we would like to have:<br>
* Tests for all python-*clients<br>
* Tests for all *-manage binaries<br>
* Tests for both admin and regular users<br>
* Read/Write tests (spin up a VM with client do something etc)<br>
<br>
<br>
<br>
<br>
<br>
<br>
Daryl, as for your WIP patch, I will take a look at it hopefully there are some things we can use from it.<br>
<br>
<br>
best,<br>
Joe Gordon<br>
<br>
<br>
On Thu, Feb 21, 2013 at 8:18 AM, Daryl Walleck < <a href="mailto:daryl.walleck@rackspace.com" target="_blank">daryl.walleck@rackspace.com</a> > wrote:<br>
<br>
<br>
Ahh, I didn't realize we an agreed upon solution for command line testing already. If there's nothing of use that can be pulled from what I submitted, I'll withdraw the branch.<br>
<br>
Daryl<br>
________________________________________<br>
From: Jay Pipes [ <a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a> ]<br>
Sent: Thursday, February 21, 2013 9:44 AM<br>
To: <a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a><br>
<br>
<br>
Subject: Re: [openstack-qa] New Client Test Suite<br>
<br>
On 02/21/2013 10:24 AM, Daryl Walleck wrote:<br>
> It took me a bit longer than I hoped, but I but a basic outline of some of the command line tests we have out in a WIP branch. If there's enough interest, I can get this merged into Tempest properly.<br>
><br>
> <a href="https://review.openstack.org/#/c/22552/" target="_blank">https://review.openstack.org/#/c/22552/</a><br>
<br>
Hi Daryl!<br>
<br>
We now have a CLI test module in Tempest:<br>
<br>
<a href="https://github.com/openstack/tempest/tree/master/cli" target="_blank">https://github.com/openstack/tempest/tree/master/cli</a><br>
<br>
If we bring in your CLI tests, I think I'd prefer to:<br>
<br>
a) Keep the code style consistent with the rest of Tempest<br>
b) Use the simple framework that Joe put together for the cli tests<br>
currently in Tempest<br>
<br>
Best,<br>
-jay<br>
<br>
> Daryl<br>
> ________________________________________<br>
> From: Sam Danes [ <a href="mailto:sam.danes@RACKSPACE.COM" target="_blank">sam.danes@RACKSPACE.COM</a> ]<br>
> Sent: Thursday, February 14, 2013 11:15 AM<br>
> To: All Things QA.<br>
> Subject: Re: [openstack-qa] New Client Test Suite<br>
><br>
> I've given Daryl a pointer to the prototype code we wrote. He'll work on getting it out to the community.<br>
><br>
> -----Original Message-----<br>
> From: Attila Fazekas [mailto: <a href="mailto:afazekas@redhat.com" target="_blank">afazekas@redhat.com</a> ]<br>
> Sent: Thursday, February 14, 2013 7:33 AM<br>
> To: All Things QA.<br>
> Subject: Re: [openstack-qa] New Client Test Suite<br>
><br>
> Very good idea!<br>
><br>
> I think it can fit into tempest repository, but in a different directory.<br>
><br>
> I would extend a scope too.<br>
><br>
> At the first Round:<br>
> - *mange command tests.<br>
> - remote execution (with ssh connection pooling) The whitebox test type (not the existing test case), partially should be converted to utilities, partially the functionality should move to the new folder.<br>
> <a href="https://github.com/openstack/tempest/blob/master/tempest/whitebox.py#L138" target="_blank">https://github.com/openstack/tempest/blob/master/tempest/whitebox.py#L138</a><br>
><br>
><br>
> ----- Original Message -----<br>
>> From: "Daryl Walleck" < <a href="mailto:daryl.walleck@RACKSPACE.COM" target="_blank">daryl.walleck@RACKSPACE.COM</a> ><br>
>> To: "All Things QA." < <a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a> ><br>
>> Sent: Thursday, February 14, 2013 3:33:20 AM<br>
>> Subject: Re: [openstack-qa] New Client Test Suite<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> We've bumped into some Novaclient only issues before in the past as<br>
>> well. Sam and some of his team did prototype some tests that execute<br>
>> using Novaclient. If there's any interest I can see about getting it<br>
>> into a WIP branch to get eyes on it.<br>
>><br>
>><br>
>><br>
>> Daryl<br>
>><br>
>><br>
>> From: Joe Gordon [ <a href="mailto:jogo@cloudscaling.com" target="_blank">jogo@cloudscaling.com</a> ]<br>
>> Sent: Tuesday, February 12, 2013 4:39 PM<br>
>> To: <a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a><br>
>> Subject: [openstack-qa] New Client Test Suite<br>
>><br>
>><br>
>><br>
>><br>
>> Hi All,<br>
>><br>
>> We currently have thorough API tests (Tempest) that do an excellent<br>
>> job of testing nova, glance, keystone, etc. But to date we don't have<br>
>> anything that tests the assorted python clients.<br>
>><br>
>><br>
>> Extensive python client tests, which would consist of mostly shell<br>
>> commands, would prevent many bugs from arising such as:<br>
>> <a href="https://bugs.launchpad.net/python-novaclient/+bug/1122958" target="_blank">https://bugs.launchpad.net/python-novaclient/+bug/1122958</a> .<br>
>><br>
>><br>
>> The question is where should theses tests live?<br>
>><br>
>><br>
>> * Devstack/smoketests - these are just smoketests and are not designed<br>
>> to be exhaustive<br>
>> * Tempest - currently explicitly doesn't use the python clients, but<br>
>> we can add a separate folder that tests the clients<br>
>> * In respective client repos - easier to require new tests when adding<br>
>> to client, but will result in divergent test suites<br>
>> * Separate Repo - perhaps this is premature?<br>
>><br>
>><br>
>> I propose adding a separate folder in tempest to run the python client<br>
>> tests, and taking it from there.<br>
>><br>
>><br>
>><br>
>><br>
>> best,<br>
>> Joe Gordon<br>
>> _______________________________________________<br>
>> openstack-qa mailing list<br>
>> <a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
>><br>
><br>
> _______________________________________________<br>
> openstack-qa mailing list<br>
> <a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
><br>
> _______________________________________________<br>
> openstack-qa mailing list<br>
> <a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
><br>
> _______________________________________________<br>
> openstack-qa mailing list<br>
> <a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
><br>
<br>
_______________________________________________<br>
openstack-qa mailing list<br>
<a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
<br>
_______________________________________________<br>
openstack-qa mailing list<br>
<a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
<br>
<br>
_______________________________________________<br>
openstack-qa mailing list<br>
<a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
<br>
_______________________________________________<br>
openstack-qa mailing list<br>
<a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
</div></div></blockquote></div><br>