<br><br><div class="gmail_quote">On Thu, Feb 21, 2013 at 11:00 AM, David Kranz <span dir="ltr"><<a href="mailto:david.kranz@qrclab.com" target="_blank">david.kranz@qrclab.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Joe, that's great. This is part of the
wonderful explosion of tests being contributed to Tempest. I am
concerned about the performance of the gate. Do we really want to
run a complete test of novaclient every time some one checks
something into swift? Or any project other than nova and
novaclient? I am not trying to single out this change but the
current situation is that a test either gates all projects, or
none. I don't see how this is sustainable. I have submitted a
summit session about this
<a href="http://summit.openstack.org/cfp/details/23" target="_blank">http://summit.openstack.org/cfp/details/23</a>.</div></div></blockquote><div><br></div><div>While I agree that this warrants further discussion. The current version of the CLI tests take only a few seconds, and as they expand (and start using testr) I don't expect the tests to make much longer than a minute or so, which I think is a reasonable amount of time to increase the gate by.</div>
<div><br></div><div>best,</div><div>Joe</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><div><span class="HOEnZb"><font color="#888888"><br>
<br>
-David</font></span><div><div class="h5"><br>
<br>
On 2/21/2013 1:47 PM, Joe Gordon wrote:<br>
</div></div></div><div><div class="h5">
<blockquote type="cite">Hi All,
<div><br>
</div>
<div>I am pleased to say that the CLI tests are now gating (<a href="https://review.openstack.org/#/c/22096/" target="_blank">https://review.openstack.org/#/c/22096/</a>).
They run at the end of the standard tempest test (example: <a href="http://logs.openstack.org/22546/5/check/gate-tempest-devstack-vm-full/4878/console.html.gz" target="_blank">http://logs.openstack.org/22546/5/check/gate-tempest-devstack-vm-full/4878/console.html.gz</a>).
<div>
<br>
</div>
<div>The current set of tests are very basic and need a lot of
work. But now that the CLI tests are self-gating it should be
easy for anyone to contribute to them. Currently we have:</div>
<div>* uses tempest.conf</div>
<div>* in unittest framework</div>
<div>* admin user only</div>
<div>* python-novaclient only</div>
<div>* read-only tests</div>
<div><br>
</div>
<div>But we would like to have:</div>
<div>* Tests for all python-*clients</div>
<div>
* Tests for all *-manage binaries</div>
<div>* Tests for both admin and regular users</div>
<div>* Read/Write tests (spin up a VM with client do something
etc)</div>
<div><br>
</div>
<div>
<div>
<div><br>
</div>
<div>Daryl, as for your WIP patch, I will take a look at it
hopefully there are some things we can use from it.</div>
<div><br>
</div>
<div>best,</div>
<div>Joe Gordon</div>
<div><br>
<div class="gmail_quote">On Thu, Feb 21, 2013 at 8:18 AM,
Daryl Walleck <span dir="ltr"><<a href="mailto:daryl.walleck@rackspace.com" target="_blank">daryl.walleck@rackspace.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Ahh,
I didn't realize we an agreed upon solution for
command line testing already. If there's nothing of
use that can be pulled from what I submitted, I'll
withdraw the branch.<br>
<br>
Daryl<br>
________________________________________<br>
From: Jay Pipes [<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>]<br>
Sent: Thursday, February 21, 2013 9:44 AM<br>
To: <a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a><br>
<div>
<div>Subject: Re: [openstack-qa] New
Client Test Suite<br>
<br>
On 02/21/2013 10:24 AM, Daryl Walleck wrote:<br>
> It took me a bit longer than I hoped, but I
but a basic outline of some of the command line
tests we have out in a WIP branch. If there's
enough interest, I can get this merged into
Tempest properly.<br>
><br>
> <a href="https://review.openstack.org/#/c/22552/" target="_blank">https://review.openstack.org/#/c/22552/</a><br>
<br>
Hi Daryl!<br>
<br>
We now have a CLI test module in Tempest:<br>
<br>
<a href="https://github.com/openstack/tempest/tree/master/cli" target="_blank">https://github.com/openstack/tempest/tree/master/cli</a><br>
<br>
If we bring in your CLI tests, I think I'd prefer
to:<br>
<br>
a) Keep the code style consistent with the rest of
Tempest<br>
b) Use the simple framework that Joe put together
for the cli tests<br>
currently in Tempest<br>
<br>
Best,<br>
-jay<br>
<br>
> Daryl<br>
> ________________________________________<br>
> From: Sam Danes [<a href="mailto:sam.danes@RACKSPACE.COM" target="_blank">sam.danes@RACKSPACE.COM</a>]<br>
> Sent: Thursday, February 14, 2013 11:15 AM<br>
> To: All Things QA.<br>
> Subject: Re: [openstack-qa] New Client Test
Suite<br>
><br>
> I've given Daryl a pointer to the prototype
code we wrote. He'll work on getting it out to the
community.<br>
><br>
> -----Original Message-----<br>
> From: Attila Fazekas [mailto:<a href="mailto:afazekas@redhat.com" target="_blank">afazekas@redhat.com</a>]<br>
> Sent: Thursday, February 14, 2013 7:33 AM<br>
> To: All Things QA.<br>
> Subject: Re: [openstack-qa] New Client Test
Suite<br>
><br>
> Very good idea!<br>
><br>
> I think it can fit into tempest repository,
but in a different directory.<br>
><br>
> I would extend a scope too.<br>
><br>
> At the first Round:<br>
> - *mange command tests.<br>
> - remote execution (with ssh connection
pooling) The whitebox test type (not the existing
test case), partially should be converted to
utilities, partially the functionality should move
to the new folder.<br>
> <a href="https://github.com/openstack/tempest/blob/master/tempest/whitebox.py#L138" target="_blank">https://github.com/openstack/tempest/blob/master/tempest/whitebox.py#L138</a><br>
><br>
><br>
> ----- Original Message -----<br>
>> From: "Daryl Walleck" <<a href="mailto:daryl.walleck@RACKSPACE.COM" target="_blank">daryl.walleck@RACKSPACE.COM</a>><br>
>> To: "All Things QA." <<a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a>><br>
>> Sent: Thursday, February 14, 2013 3:33:20
AM<br>
>> Subject: Re: [openstack-qa] New Client
Test Suite<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> We've bumped into some Novaclient only
issues before in the past as<br>
>> well. Sam and some of his team did
prototype some tests that execute<br>
>> using Novaclient. If there's any interest
I can see about getting it<br>
>> into a WIP branch to get eyes on it.<br>
>><br>
>><br>
>><br>
>> Daryl<br>
>><br>
>><br>
>> From: Joe Gordon [<a href="mailto:jogo@cloudscaling.com" target="_blank">jogo@cloudscaling.com</a>]<br>
>> Sent: Tuesday, February 12, 2013 4:39 PM<br>
>> To: <a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a><br>
>> Subject: [openstack-qa] New Client Test
Suite<br>
>><br>
>><br>
>><br>
>><br>
>> Hi All,<br>
>><br>
>> We currently have thorough API tests
(Tempest) that do an excellent<br>
>> job of testing nova, glance, keystone,
etc. But to date we don't have<br>
>> anything that tests the assorted python
clients.<br>
>><br>
>><br>
>> Extensive python client tests, which
would consist of mostly shell<br>
>> commands, would prevent many bugs from
arising such as:<br>
>> <a href="https://bugs.launchpad.net/python-novaclient/+bug/1122958" target="_blank">https://bugs.launchpad.net/python-novaclient/+bug/1122958</a>
.<br>
>><br>
>><br>
>> The question is where should theses tests
live?<br>
>><br>
>><br>
>> * Devstack/smoketests - these are just
smoketests and are not designed<br>
>> to be exhaustive<br>
>> * Tempest - currently explicitly doesn't
use the python clients, but<br>
>> we can add a separate folder that tests
the clients<br>
>> * In respective client repos - easier to
require new tests when adding<br>
>> to client, but will result in divergent
test suites<br>
>> * Separate Repo - perhaps this is
premature?<br>
>><br>
>><br>
>> I propose adding a separate folder in
tempest to run the python client<br>
>> tests, and taking it from there.<br>
>><br>
>><br>
>><br>
>><br>
>> best,<br>
>> Joe Gordon<br>
>>
_______________________________________________<br>
>> openstack-qa mailing list<br>
>> <a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
>><br>
><br>
>
_______________________________________________<br>
> openstack-qa mailing list<br>
> <a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
><br>
>
_______________________________________________<br>
> openstack-qa mailing list<br>
> <a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
><br>
>
_______________________________________________<br>
> openstack-qa mailing list<br>
> <a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
><br>
<br>
_______________________________________________<br>
openstack-qa mailing list<br>
<a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
<br>
_______________________________________________<br>
openstack-qa mailing list<br>
<a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
openstack-qa mailing list
<a href="mailto:openstack-qa@lists.openstack.org" target="_blank">openstack-qa@lists.openstack.org</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a>
</pre>
</blockquote>
<br>
</div></div></div>
<br>_______________________________________________<br>
openstack-qa mailing list<br>
<a href="mailto:openstack-qa@lists.openstack.org">openstack-qa@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
<br></blockquote></div><br>