<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Ivan,<br>
<br>
I agree that our testing needs improvement. Thanks for starting
this thread.<br>
<br>
With regards to adding a hacking check for tests that run too long
... are you thinking that we would have a timer that checks or long
running jobs or something that checks for long sleeps in the testing
code? Just curious your ideas for tackling that situation. Would
be interested in helping with that, perhaps.<br>
<br>
Thanks!<br>
Jay<br>
<br>
<div class="moz-cite-prefix">On 03/02/2016 05:25 AM, Ivan
Kolodyazhny wrote:<br>
</div>
<blockquote
cite="mid:CAGocpaH_-09tiHNDgabhZZywScRajouyVAYRgaLru3vv6x1rjw@mail.gmail.com"
type="cite">
<div dir="ltr">Hi Team,
<div><br>
</div>
<div>Here are my thoughts and proposals how to make Cinder
testing process better. I won't cover "3rd party CI's" topic
here. I will share my opinion about current and feature jobs.</div>
<div><br>
</div>
<div><br>
</div>
<div>Unit-tests</div>
<div>
<ul>
<li>Long-running tests. I hope, everybody will agree that
unit-tests must be quite simple and very fast. Unit tests
which takes more than 3-5 seconds should be refactored
and/or moved to 'integration' tests. <br>
Thanks to Tom Barron for several fixes like [1]. IMO, we
it would be good to have some hacking checks to prevent
such issues in a future.<br>
<br>
</li>
<li>Tests coverage. We don't check it in an automatic way on
gates. Usually, we require to add some unit-tests during
code review process. Why can't we add coverage job to our
CI and do not merge new patches, with will decrease tests
coverage rate? Maybe, such job could be voting in a future
to not ignore it. For now, there is not simple way to
check coverage because 'tox -e cover' output is not useful
[2].<br>
</li>
</ul>
</div>
<div><br>
</div>
<div>Functional tests for Cinder</div>
<div><br>
</div>
<div>We introduced some functional tests last month [3]. Here is
a patch to infra to add new job [4]. Because these tests were
moved from unit-tests, I think we're OK to make this job
voting. Such tests should not be a replacement for Tempest.
They even could tests Cinder with Fake Driver to make it
faster and not related on storage backends issues.</div>
<div><br>
</div>
<div><br>
</div>
<div>Tempest in-tree tests</div>
<div><br>
</div>
<div>Sean started work on it [5] and I think it's a good idea to
get them in Cinder repo to run them on Tempest jobs and 3-rd
party CIs against a real backend.</div>
<div><br>
</div>
<div><br>
</div>
Functional tests for python-brick-cinderclient-ext
<div><br>
</div>
<div>There are patches that introduces functional tests [6] and
new job [7].</div>
<div><br>
</div>
<div><br>
</div>
<div>Functional tests for python-cinderclient</div>
<div><br>
</div>
<div>We've got a very limited set of such tests and non-voting
job. IMO, we can run them even with Cinder Fake Driver to make
them not depended on a storage backend and make it faster. I
believe, we can make this job voting soon. Also, we need more
contributors to this kind of tests.</div>
<div><br>
</div>
<div><br>
</div>
<div>Integrated tests for python-cinderclient</div>
<div><br>
</div>
<div>We need such tests to make sure that we won't break Nova,
Heat or other python-cinderclient consumers with a next merged
patch. There is a thread in openstack-dev ML about such tests
[8] and proposal [9] to introduce them to python-cinderclient.</div>
<div><br>
</div>
<div><br>
</div>
<div>Rally tests</div>
<div><br>
</div>
<div>IMO, it would be good to have new Rally scenarios for every
patches like 'improves performance', 'fixes concurrency
issues', etc. Even if we as a Cinder community don't have
enough time to implement them, we have to ask for them in
reviews, openstack-dev ML, file Rally bugs and blueprints if
needed.</div>
<div><br>
</div>
<div><br>
</div>
<div>[1] <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/282861/">https://review.openstack.org/#/c/282861/</a></div>
<div>[2] <a moz-do-not-send="true"
href="http://paste.openstack.org/show/488925/">http://paste.openstack.org/show/488925/</a></div>
<div>[3] <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/267801/">https://review.openstack.org/#/c/267801/</a></div>
<div>[4] <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/287115/">https://review.openstack.org/#/c/287115/</a></div>
<div>[5] <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/274471/">https://review.openstack.org/#/c/274471/</a></div>
<div>[6] <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/265811/">https://review.openstack.org/#/c/265811/</a></div>
<div>[7] <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/265925/">https://review.openstack.org/#/c/265925/</a></div>
<div>[8] <a moz-do-not-send="true"
href="http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html">http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html</a></div>
<div>[9] <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/279432/">https://review.openstack.org/#/c/279432/</a></div>
<div><br>
</div>
<div><br clear="all">
<div>
<div class="gmail_signature">
<div dir="ltr">
<div>Regards,<br>
Ivan Kolodyazhny,<br>
<a moz-do-not-send="true"
href="http://blog.e0ne.info/" target="_blank">http://blog.e0ne.info/</a></div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>