<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.im
{mso-style-name:im;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Rally is not part of the gate.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Also making performance test without 3<sup>rd</sup> party CI will not be very useful.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">It is a good idea to run Rally performance and scenario testing but outside gate process.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><a name="_____replyseparator"></a><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Ivan Kolodyazhny [mailto:e0ne@e0ne.info]
<br>
<b>Sent:</b> Wednesday, March 02, 2016 8:36 AM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org><br>
<b>Subject:</b> Re: [openstack-dev] [cinder] Proposal: changes to our current testing process<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Eric,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">There are Gorka's patches [10] to remove API Races<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">[10] <a href="https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified">https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified</a><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Regards,<br>
Ivan Kolodyazhny,<br>
<a href="http://blog.e0ne.info/" target="_blank">http://blog.e0ne.info/</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney <<a href="mailto:eharney@redhat.com" target="_blank">eharney@redhat.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:<br>
> Hi Team,<br>
><br>
> Here are my thoughts and proposals how to make Cinder testing process<br>
> better. I won't cover "3rd party CI's" topic here. I will share my opinion<br>
> about current and feature jobs.<br>
><br>
><br>
> Unit-tests<br>
><br>
> - Long-running tests. I hope, everybody will agree that unit-tests must<br>
> be quite simple and very fast. Unit tests which takes more than 3-5 seconds<br>
> should be refactored and/or moved to 'integration' tests.<br>
> Thanks to Tom Barron for several fixes like [1]. IMO, we it would be<br>
> good to have some hacking checks to prevent such issues in a future.<br>
><br>
> - Tests coverage. We don't check it in an automatic way on gates.<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">> Usually, we require to add some unit-tests during code review process. Why<br>
> can't we add coverage job to our CI and do not merge new patches, with<br>
> will decrease tests coverage rate? Maybe, such job could be voting in a<br>
> future to not ignore it. For now, there is not simple way to check coverage<br>
> because 'tox -e cover' output is not useful [2].<br>
><br>
><br>
> Functional tests for Cinder<br>
><br>
> We introduced some functional tests last month [3]. Here is a patch to<br>
> infra to add new job [4]. Because these tests were moved from unit-tests, I<br>
> think we're OK to make this job voting. Such tests should not be a<br>
> replacement for Tempest. They even could tests Cinder with Fake Driver to<br>
> make it faster and not related on storage backends issues.<br>
><br>
><br>
> Tempest in-tree tests<br>
><br>
> Sean started work on it [5] and I think it's a good idea to get them in<br>
> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real<br>
> backend.<br>
><br>
><br>
> Functional tests for python-brick-cinderclient-ext<br>
><br>
> There are patches that introduces functional tests [6] and new job [7].<br>
><br>
><br>
> Functional tests for python-cinderclient<br>
><br>
> We've got a very limited set of such tests and non-voting job. IMO, we can<br>
> run them even with Cinder Fake Driver to make them not depended on a<br>
> storage backend and make it faster. I believe, we can make this job voting<br>
> soon. Also, we need more contributors to this kind of tests.<br>
><br>
><br>
> Integrated tests for python-cinderclient<br>
><br>
> We need such tests to make sure that we won't break Nova, Heat or other<br>
> python-cinderclient consumers with a next merged patch. There is a thread<br>
> in openstack-dev ML about such tests [8] and proposal [9] to introduce them<br>
> to python-cinderclient.<br>
><br>
><br>
> Rally tests<br>
><br>
> IMO, it would be good to have new Rally scenarios for every patches like<br>
> 'improves performance', 'fixes concurrency issues', etc. Even if we as a<br>
> Cinder community don't have enough time to implement them, we have to ask<br>
> for them in reviews, openstack-dev ML, file Rally bugs and blueprints if<br>
> needed.<br>
><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal">Are there any recent examples of a fix like this recently where it would<br>
seem like a reasonable task to write a Rally scenario along with the patch?<br>
<br>
Not being very familiar with Rally (as I think most of us aren't), I'm<br>
having a hard time picturing this.<br>
<br>
<span class="im">></span><br>
<span class="im">> [1] <a href="https://review.openstack.org/#/c/282861/" target="_blank">
https://review.openstack.org/#/c/282861/</a></span><br>
<span class="im">> [2] <a href="http://paste.openstack.org/show/488925/" target="_blank">
http://paste.openstack.org/show/488925/</a></span><br>
<span class="im">> [3] <a href="https://review.openstack.org/#/c/267801/" target="_blank">
https://review.openstack.org/#/c/267801/</a></span><br>
<span class="im">> [4] <a href="https://review.openstack.org/#/c/287115/" target="_blank">
https://review.openstack.org/#/c/287115/</a></span><br>
<span class="im">> [5] <a href="https://review.openstack.org/#/c/274471/" target="_blank">
https://review.openstack.org/#/c/274471/</a></span><br>
<span class="im">> [6] <a href="https://review.openstack.org/#/c/265811/" target="_blank">
https://review.openstack.org/#/c/265811/</a></span><br>
<span class="im">> [7] <a href="https://review.openstack.org/#/c/265925/" target="_blank">
https://review.openstack.org/#/c/265925/</a></span><br>
<span class="im">> [8]</span><br>
<span class="im">> <a href="http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html" target="_blank">
http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html</a></span><br>
<span class="im">> [9] <a href="https://review.openstack.org/#/c/279432/" target="_blank">
https://review.openstack.org/#/c/279432/</a></span><br>
<span class="im">></span><br>
<span class="im">></span><br>
<span class="im">> Regards,</span><br>
<span class="im">> Ivan Kolodyazhny,</span><br>
<span class="im">> <a href="http://blog.e0ne.info/" target="_blank">http://blog.e0ne.info/</a></span><br>
<span class="im">></span><br>
<span class="im">></span><br>
<span class="im">></span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>