<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mv="http://macVmlSchemaUri" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Arial;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman";}
span.EmailStyle18
{mso-style-type:personal;
font-family:Calibri;
color:#1F497D;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Nate, you have to press Ctrl-C to see the in-progress test, that’s why you don’t see it in the logs. The bug report shows this and points to the patch where it appeared to begin.
<a href="https://bugs.launchpad.net/cinder/+bug/1578986">https://bugs.launchpad.net/cinder/+bug/1578986</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Clinton<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black">"Potter, Nathaniel" <nathaniel.potter@intel.com><br>
<b>Reply-To: </b>"OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org><br>
<b>Date: </b>Thursday, July 21, 2016 at 7:17 PM<br>
<b>To: </b>"OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org><br>
<b>Subject: </b>Re: [openstack-dev] [cinder] Volume Drivers unit tests<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;color:#1F497D">Hi all,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;color:#1F497D">I’m not totally sure that this is the same issue, but lately I’ve seen the gate tests fail while hanging at this point [1], but they say ‘ok’ rather than ‘inprogress’. Has
anyone else come across this? It only happens sometimes, and a recheck can get past it. The full log is here [2].</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;color:#1F497D">[1]
<a href="http://paste.openstack.org/show/539314/">http://paste.openstack.org/show/539314/</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;color:#1F497D">[2]
<a href="http://logs.openstack.org/90/341090/6/check/gate-cinder-python34-db/ea65de5/console.html">
http://logs.openstack.org/90/341090/6/check/gate-cinder-python34-db/ea65de5/console.html</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;color:#1F497D">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;color:#1F497D">Nate</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><a name="_____replyseparator"></a><b><span style="font-size:11.0pt;font-family:Calibri">From:</span></b><span style="font-size:11.0pt;font-family:Calibri"> yang, xing [mailto:xing.yang@emc.com]
<br>
<b>Sent:</b> Thursday, July 21, 2016 3:17 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org><br>
<b>Subject:</b> Re: [openstack-dev] [cinder] Volume Drivers unit tests</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Tahoma;color:black">Hi Ivan,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Tahoma;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Tahoma;color:black">Do you have any logs for the VMAX driver? We'll take a look.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Tahoma;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Tahoma;color:black">Thanks,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Tahoma;color:black">Xing</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Tahoma;color:black"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Tahoma;color:black"> </span><o:p></o:p></p>
<div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="color:black">
<hr size="3" width="100%" align="center">
</span></div>
<div id="divRpF880066">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:10.0pt;font-family:Tahoma;color:black">From:</span></b><span style="font-size:10.0pt;font-family:Tahoma;color:black"> Ivan Kolodyazhny [e0ne@e0ne.info]<br>
<b>Sent:</b> Thursday, July 21, 2016 4:44 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [cinder] Volume Drivers unit tests</span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="color:black">Thank you Xing, </span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">The issue is related both to VNX and VMAX EMC drivers</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><br clear="all">
</span><o:p></o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:black">Regards,<br>
Ivan Kolodyazhny,<br>
<a href="http://blog.e0ne.info/" target="_blank">http://blog.e0ne.info/</a></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="color:black">On Thu, Jul 21, 2016 at 11:00 PM, yang, xing <<a href="mailto:xing.yang@emc.com" target="_blank">xing.yang@emc.com</a>> wrote:</span><o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Tahoma;color:black">Hi Ivan,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Tahoma;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Tahoma;color:black">Thanks for sending this out. Regarding the issue in the EMC VNX driver unit tests, it is tracked by this bug <a href="https://bugs.launchpad.net/cinder/+bug/1578986" target="_blank">https://bugs.launchpad.net/cinder/+bug/1578986</a>.
The driver was recently refactored so this is probably a new issue introduced by the refactor. We are investigating this issue.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Tahoma;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Tahoma;color:black">Thanks,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Tahoma;color:black">Xing</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Tahoma;color:black"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Tahoma;color:black"> </span><o:p></o:p></p>
<div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="color:black">
<hr size="3" width="100%" align="center">
</span></div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:10.0pt;font-family:Tahoma;color:black">From:</span></b><span style="font-size:10.0pt;font-family:Tahoma;color:black"> Ivan Kolodyazhny [<a href="mailto:e0ne@e0ne.info" target="_blank">e0ne@e0ne.info</a>]<br>
<b>Sent:</b> Thursday, July 21, 2016 1:02 PM<br>
<b>To:</b> OpenStack Development Mailing List<br>
<b>Subject:</b> [openstack-dev] [cinder] Volume Drivers unit tests</span><o:p></o:p></p>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:black">Hi team, </span><o:p></o:p></p>
<div>
<p><span style="color:black">First of all, I would like to apologize, if my mail is be too emotional. I spent too much of time to fix it and failed.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">TL;DR;</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">What I want to say is: "Let's spend some time to make our tests better and fix all issues". Patch [1] is still unstable. Unit tests can pass or fail in a in a random order. Also, I've disabled some tests to pass
CI.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Long version:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">While I was working on patch "Move drivers unit tests to unit.volume.drivers directory" [1] I've found a lot of issues with our unit tests :(. Not all of them are already fixed, so that patch is still in progress</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">What did I found and what should we have to fix:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">1) Execution time [2]. I don't want to argue what it unit tests, but 2-4 seconds per tests should be non-acceptable, IMO.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">2) Execution order. Seriously, do you know that our tests will fail or hang if execution order will change? Even if one test for diver A failed, some tests for driver B will fail too.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">3) Lack of mock. It's a root cause for #2. We didn't mock sleeps and event loops right. We don't mock RPC call well too [3]. We don't have 'cinder.openstack.common.rpc.impl_fake' module in Cinder tree.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">In some drivers, we use oslo_service.loopingcall.FixedIntervalLoopingCall [4]. We've go ZeroIntervalLoopingCall [5] class in Cinder. Do we use it everywhere or mock FixedIntervalLoopingCall right? I don't think
so, I've hacked oslo_service in my env to rise an exception if interval > 0. 297 tests failed. It means, our tests use sleep. We have to get rid of this. TBH, not only volume drivers unit tests failed. E.g. some API unit tests failed too.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">4) Due to #3, sometimes unit tests hangs even on master branch with a minor changes.If I stop execution of such tests, usually I see something like [6]. In most of cases I see that following drivers' tests hangs:
EMC, Huawei, Dell and RBD.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">It's hard to debug such failures because the lack of tooling for eventlet debugging. Eventlet backdoors and gdb-python helps a bit. Maybe somebody know better solution for it.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">[1] <a href="https://review.openstack.org/#/c/320148/" target="_blank">https://review.openstack.org/#/c/320148/</a></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">[2] <a href="http://paste.openstack.org/show/539081/" target="_blank">http://paste.openstack.org/show/539081/</a></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">[3] <a href="https://github.com/openstack/cinder/search?utf8=%E2%9C%93&q=impl_fake" target="_blank">https://github.com/openstack/cinder/search?utf8=%E2%9C%93&q=impl_fake</a></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">[4] use <a href="https://github.com/openstack/oslo.service/blob/master/oslo_service/loopingcall.py#L162" target="_blank">https://github.com/openstack/oslo.service/blob/master/oslo_service/loopingcall.py#L162</a></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">[5] <a href="https://github.com/openstack/cinder/blob/cfbb5bde4d9b37c39f6813fe685f987f8a990483/cinder/tests/unit/utils.py#L289" target="_blank">https://github.com/openstack/cinder/blob/cfbb5bde4d9b37c39f6813fe685f987f8a990483/cinder/tests/unit/utils.py#L289</a></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">[6] <a href="http://paste.openstack.org/show/539090/" target="_blank">http://paste.openstack.org/show/539090/</a></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><br clear="all">
</span><o:p></o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:black">Regards,<br>
Ivan Kolodyazhny,<br>
<a href="http://blog.e0ne.info/" target="_blank">http://blog.e0ne.info/</a></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:black"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></span><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>