<div dir="ltr">Arkady, <br><div><br></div>It's not true. We've got non-voting Rally job on Cinder gates called "gate-rally-dsvm-cinder".</div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div>Regards,<br>Ivan Kolodyazhny,<br><a href="http://blog.e0ne.info/" target="_blank">http://blog.e0ne.info/</a></div></div></div></div>
<br><div class="gmail_quote">On Wed, Mar 2, 2016 at 4:52 PM, <span dir="ltr"><<a href="mailto:Arkady_Kanevsky@dell.com" target="_blank">Arkady_Kanevsky@dell.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Rally is not part of the gate.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Also making performance test without 3<sup>rd</sup> party CI will not be very useful.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">It is a good idea to run Rally performance and scenario testing but outside gate process.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><a name="1829299799______replyseparator"></a><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Ivan Kolodyazhny [mailto:<a href="mailto:e0ne@e0ne.info" target="_blank">e0ne@e0ne.info</a>]
<br>
<b>Sent:</b> Wednesday, March 02, 2016 8:36 AM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions) <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<b>Subject:</b> Re: [openstack-dev] [cinder] Proposal: changes to our current testing process<u></u><u></u></span></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Eric,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">There are Gorka's patches [10] to remove API Races<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">[10] <a href="https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified" target="_blank">https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified</a><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Regards,<br>
Ivan Kolodyazhny,<br>
<a href="http://blog.e0ne.info/" target="_blank">http://blog.e0ne.info/</a><u></u><u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney <<a href="mailto:eharney@redhat.com" target="_blank">eharney@redhat.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:<br>
> Hi Team,<br>
><br>
> Here are my thoughts and proposals how to make Cinder testing process<br>
> better. I won't cover "3rd party CI's" topic here. I will share my opinion<br>
> about current and feature jobs.<br>
><br>
><br>
> Unit-tests<br>
><br>
> - Long-running tests. I hope, everybody will agree that unit-tests must<br>
> be quite simple and very fast. Unit tests which takes more than 3-5 seconds<br>
> should be refactored and/or moved to 'integration' tests.<br>
> Thanks to Tom Barron for several fixes like [1]. IMO, we it would be<br>
> good to have some hacking checks to prevent such issues in a future.<br>
><br>
> - Tests coverage. We don't check it in an automatic way on gates.<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">> Usually, we require to add some unit-tests during code review process. Why<br>
> can't we add coverage job to our CI and do not merge new patches, with<br>
> will decrease tests coverage rate? Maybe, such job could be voting in a<br>
> future to not ignore it. For now, there is not simple way to check coverage<br>
> because 'tox -e cover' output is not useful [2].<br>
><br>
><br>
> Functional tests for Cinder<br>
><br>
> We introduced some functional tests last month [3]. Here is a patch to<br>
> infra to add new job [4]. Because these tests were moved from unit-tests, I<br>
> think we're OK to make this job voting. Such tests should not be a<br>
> replacement for Tempest. They even could tests Cinder with Fake Driver to<br>
> make it faster and not related on storage backends issues.<br>
><br>
><br>
> Tempest in-tree tests<br>
><br>
> Sean started work on it [5] and I think it's a good idea to get them in<br>
> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real<br>
> backend.<br>
><br>
><br>
> Functional tests for python-brick-cinderclient-ext<br>
><br>
> There are patches that introduces functional tests [6] and new job [7].<br>
><br>
><br>
> Functional tests for python-cinderclient<br>
><br>
> We've got a very limited set of such tests and non-voting job. IMO, we can<br>
> run them even with Cinder Fake Driver to make them not depended on a<br>
> storage backend and make it faster. I believe, we can make this job voting<br>
> soon. Also, we need more contributors to this kind of tests.<br>
><br>
><br>
> Integrated tests for python-cinderclient<br>
><br>
> We need such tests to make sure that we won't break Nova, Heat or other<br>
> python-cinderclient consumers with a next merged patch. There is a thread<br>
> in openstack-dev ML about such tests [8] and proposal [9] to introduce them<br>
> to python-cinderclient.<br>
><br>
><br>
> Rally tests<br>
><br>
> IMO, it would be good to have new Rally scenarios for every patches like<br>
> 'improves performance', 'fixes concurrency issues', etc. Even if we as a<br>
> Cinder community don't have enough time to implement them, we have to ask<br>
> for them in reviews, openstack-dev ML, file Rally bugs and blueprints if<br>
> needed.<br>
><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal">Are there any recent examples of a fix like this recently where it would<br>
seem like a reasonable task to write a Rally scenario along with the patch?<br>
<br>
Not being very familiar with Rally (as I think most of us aren't), I'm<br>
having a hard time picturing this.<br>
<br>
<span>></span><br>
<span>> [1] <a href="https://review.openstack.org/#/c/282861/" target="_blank">
https://review.openstack.org/#/c/282861/</a></span><br>
<span>> [2] <a href="http://paste.openstack.org/show/488925/" target="_blank">
http://paste.openstack.org/show/488925/</a></span><br>
<span>> [3] <a href="https://review.openstack.org/#/c/267801/" target="_blank">
https://review.openstack.org/#/c/267801/</a></span><br>
<span>> [4] <a href="https://review.openstack.org/#/c/287115/" target="_blank">
https://review.openstack.org/#/c/287115/</a></span><br>
<span>> [5] <a href="https://review.openstack.org/#/c/274471/" target="_blank">
https://review.openstack.org/#/c/274471/</a></span><br>
<span>> [6] <a href="https://review.openstack.org/#/c/265811/" target="_blank">
https://review.openstack.org/#/c/265811/</a></span><br>
<span>> [7] <a href="https://review.openstack.org/#/c/265925/" target="_blank">
https://review.openstack.org/#/c/265925/</a></span><br>
<span>> [8]</span><br>
<span>> <a href="http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html" target="_blank">
http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html</a></span><br>
<span>> [9] <a href="https://review.openstack.org/#/c/279432/" target="_blank">
https://review.openstack.org/#/c/279432/</a></span><br>
<span>></span><br>
<span>></span><br>
<span>> Regards,</span><br>
<span>> Ivan Kolodyazhny,</span><br>
<span>> <a href="http://blog.e0ne.info/" target="_blank">http://blog.e0ne.info/</a></span><br>
<span>></span><br>
<span>></span><br>
<span>></span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>