<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Consolas","serif";
color:black;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";
color:black;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">+1 – we need some discussion on this as services has grown.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> David Kranz [mailto:david.kranz@qrclab.com]
<br>
<b>Sent:</b> Thursday, February 21, 2013 11:00 AM<br>
<b>To:</b> openstack-qa@lists.openstack.org<br>
<b>Subject:</b> Re: [openstack-qa] New Client Test Suite<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Joe, that's great. This is part of the wonderful explosion of tests being contributed to Tempest. I am concerned about the performance of the gate. Do we really want to run a complete test of novaclient every time some one checks something
into swift? Or any project other than nova and novaclient? I am not trying to single out this change but the current situation is that a test either gates all projects, or none. I don't see how this is sustainable. I have submitted a summit session about this
<a href="http://summit.openstack.org/cfp/details/23">http://summit.openstack.org/cfp/details/23</a>.<br>
<br>
-David<br>
<br>
On 2/21/2013 1:47 PM, Joe Gordon wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Hi All, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I am pleased to say that the CLI tests are now gating (<a href="https://review.openstack.org/#/c/22096/">https://review.openstack.org/#/c/22096/</a>). They run at the end of the standard tempest test (example: <a href="http://logs.openstack.org/22546/5/check/gate-tempest-devstack-vm-full/4878/console.html.gz">http://logs.openstack.org/22546/5/check/gate-tempest-devstack-vm-full/4878/console.html.gz</a>).
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The current set of tests are very basic and need a lot of work. But now that the CLI tests are self-gating it should be easy for anyone to contribute to them. Currently we have:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">* uses tempest.conf<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">* in unittest framework<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">* admin user only<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">* python-novaclient only<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">* read-only tests<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">But we would like to have:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">* Tests for all python-*clients<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">* Tests for all *-manage binaries<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">* Tests for both admin and regular users<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">* Read/Write tests (spin up a VM with client do something etc)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Daryl, as for your WIP patch, I will take a look at it hopefully there are some things we can use from it.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">best,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Joe Gordon<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Thu, Feb 21, 2013 at 8:18 AM, Daryl Walleck <<a href="mailto:daryl.walleck@rackspace.com" target="_blank">daryl.walleck@rackspace.com</a>> wrote:<o:p></o:p></p>
<p class="MsoNormal">Ahh, I didn't realize we an agreed upon solution for command line testing already. If there's nothing of use that can be pulled from what I submitted, I'll withdraw the branch.<br>
<br>
Daryl<br>
________________________________________<br>
From: Jay Pipes [<a href="mailto:jaypipes@gmail.com">jaypipes@gmail.com</a>]<br>
Sent: Thursday, February 21, 2013 9:44 AM<br>
To: <a href="mailto:openstack-qa@lists.openstack.org">openstack-qa@lists.openstack.org</a><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">Subject: Re: [openstack-qa] New Client Test Suite<br>
<br>
On 02/21/2013 10:24 AM, Daryl Walleck wrote:<br>
> It took me a bit longer than I hoped, but I but a basic outline of some of the command line tests we have out in a WIP branch. If there's enough interest, I can get this merged into Tempest properly.<br>
><br>
> <a href="https://review.openstack.org/#/c/22552/" target="_blank">https://review.openstack.org/#/c/22552/</a><br>
<br>
Hi Daryl!<br>
<br>
We now have a CLI test module in Tempest:<br>
<br>
<a href="https://github.com/openstack/tempest/tree/master/cli" target="_blank">https://github.com/openstack/tempest/tree/master/cli</a><br>
<br>
If we bring in your CLI tests, I think I'd prefer to:<br>
<br>
a) Keep the code style consistent with the rest of Tempest<br>
b) Use the simple framework that Joe put together for the cli tests<br>
currently in Tempest<br>
<br>
Best,<br>
-jay<br>
<br>
> Daryl<br>
> ________________________________________<br>
> From: Sam Danes [<a href="mailto:sam.danes@RACKSPACE.COM">sam.danes@RACKSPACE.COM</a>]<br>
> Sent: Thursday, February 14, 2013 11:15 AM<br>
> To: All Things QA.<br>
> Subject: Re: [openstack-qa] New Client Test Suite<br>
><br>
> I've given Daryl a pointer to the prototype code we wrote. He'll work on getting it out to the community.<br>
><br>
> -----Original Message-----<br>
> From: Attila Fazekas [mailto:<a href="mailto:afazekas@redhat.com">afazekas@redhat.com</a>]<br>
> Sent: Thursday, February 14, 2013 7:33 AM<br>
> To: All Things QA.<br>
> Subject: Re: [openstack-qa] New Client Test Suite<br>
><br>
> Very good idea!<br>
><br>
> I think it can fit into tempest repository, but in a different directory.<br>
><br>
> I would extend a scope too.<br>
><br>
> At the first Round:<br>
> - *mange command tests.<br>
> - remote execution (with ssh connection pooling) The whitebox test type (not the existing test case), partially should be converted to utilities, partially the functionality should move to the new folder.<br>
> <a href="https://github.com/openstack/tempest/blob/master/tempest/whitebox.py#L138" target="_blank">
https://github.com/openstack/tempest/blob/master/tempest/whitebox.py#L138</a><br>
><br>
><br>
> ----- Original Message -----<br>
>> From: "Daryl Walleck" <<a href="mailto:daryl.walleck@RACKSPACE.COM">daryl.walleck@RACKSPACE.COM</a>><br>
>> To: "All Things QA." <<a href="mailto:openstack-qa@lists.openstack.org">openstack-qa@lists.openstack.org</a>><br>
>> Sent: Thursday, February 14, 2013 3:33:20 AM<br>
>> Subject: Re: [openstack-qa] New Client Test Suite<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> We've bumped into some Novaclient only issues before in the past as<br>
>> well. Sam and some of his team did prototype some tests that execute<br>
>> using Novaclient. If there's any interest I can see about getting it<br>
>> into a WIP branch to get eyes on it.<br>
>><br>
>><br>
>><br>
>> Daryl<br>
>><br>
>><br>
>> From: Joe Gordon [<a href="mailto:jogo@cloudscaling.com">jogo@cloudscaling.com</a>]<br>
>> Sent: Tuesday, February 12, 2013 4:39 PM<br>
>> To: <a href="mailto:openstack-qa@lists.openstack.org">openstack-qa@lists.openstack.org</a><br>
>> Subject: [openstack-qa] New Client Test Suite<br>
>><br>
>><br>
>><br>
>><br>
>> Hi All,<br>
>><br>
>> We currently have thorough API tests (Tempest) that do an excellent<br>
>> job of testing nova, glance, keystone, etc. But to date we don't have<br>
>> anything that tests the assorted python clients.<br>
>><br>
>><br>
>> Extensive python client tests, which would consist of mostly shell<br>
>> commands, would prevent many bugs from arising such as:<br>
>> <a href="https://bugs.launchpad.net/python-novaclient/+bug/1122958" target="_blank">
https://bugs.launchpad.net/python-novaclient/+bug/1122958</a> .<br>
>><br>
>><br>
>> The question is where should theses tests live?<br>
>><br>
>><br>
>> * Devstack/smoketests - these are just smoketests and are not designed<br>
>> to be exhaustive<br>
>> * Tempest - currently explicitly doesn't use the python clients, but<br>
>> we can add a separate folder that tests the clients<br>
>> * In respective client repos - easier to require new tests when adding<br>
>> to client, but will result in divergent test suites<br>
>> * Separate Repo - perhaps this is premature?<br>
>><br>
>><br>
>> I propose adding a separate folder in tempest to run the python client<br>
>> tests, and taking it from there.<br>
>><br>
>><br>
>><br>
>><br>
>> best,<br>
>> Joe Gordon<br>
>> _______________________________________________<br>
>> openstack-qa mailing list<br>
>> <a href="mailto:openstack-qa@lists.openstack.org">openstack-qa@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
>><br>
><br>
> _______________________________________________<br>
> openstack-qa mailing list<br>
> <a href="mailto:openstack-qa@lists.openstack.org">openstack-qa@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
><br>
> _______________________________________________<br>
> openstack-qa mailing list<br>
> <a href="mailto:openstack-qa@lists.openstack.org">openstack-qa@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
><br>
> _______________________________________________<br>
> openstack-qa mailing list<br>
> <a href="mailto:openstack-qa@lists.openstack.org">openstack-qa@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
><br>
<br>
_______________________________________________<br>
openstack-qa mailing list<br>
<a href="mailto:openstack-qa@lists.openstack.org">openstack-qa@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><br>
<br>
_______________________________________________<br>
openstack-qa mailing list<br>
<a href="mailto:openstack-qa@lists.openstack.org">openstack-qa@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>openstack-qa mailing list<o:p></o:p></pre>
<pre><a href="mailto:openstack-qa@lists.openstack.org">openstack-qa@lists.openstack.org</a><o:p></o:p></pre>
<pre><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>