<div dir="ltr">Thank you Xing,<div><br></div><div>The issue is related both to VNX and VMAX EMC drivers</div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Regards,<br>Ivan Kolodyazhny,<br><a href="http://blog.e0ne.info/" target="_blank">http://blog.e0ne.info/</a></div></div></div></div>
<br><div class="gmail_quote">On Thu, Jul 21, 2016 at 11:00 PM, yang, xing <span dir="ltr"><<a href="mailto:xing.yang@emc.com" target="_blank">xing.yang@emc.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">
<div>Hi Ivan,</div>
<div><br>
</div>
<div>Thanks for sending this out. Regarding the issue in the EMC VNX driver unit tests, it is tracked by this bug <a href="https://bugs.launchpad.net/cinder/+bug/1578986" target="_blank">https://bugs.launchpad.net/cinder/+bug/1578986</a>. The driver was recently
refactored so this is probably a new issue introduced by the refactor. We <span style="font-size:10pt">are investigating this issue.</span></div>
<div><br>
</div>
<div>Thanks,</div>
<div>Xing</div>
<div><br>
</div>
<br>
<div style="font-family:Times New Roman;color:#000000;font-size:16px">
<hr>
<div style="direction:ltr"><font face="Tahoma" size="2" color="#000000"><b>From:</b> Ivan Kolodyazhny [<a href="mailto:e0ne@e0ne.info" target="_blank">e0ne@e0ne.info</a>]<br>
<b>Sent:</b> Thursday, July 21, 2016 1:02 PM<br>
<b>To:</b> OpenStack Development Mailing List<br>
<b>Subject:</b> [openstack-dev] [cinder] Volume Drivers unit tests<br>
</font><br>
</div><div><div class="h5">
<div></div>
<div>
<div dir="ltr">Hi team,
<div>
<p><span>First of all, I would like to apologize, if my mail is</span><span> be too emotional. I spent too much of time to fix it and failed.</span></p>
TL;DR;</div>
<div><br>
</div>
<div>What I want to say is: "Let's spend some time to make our tests better and fix all issues". Patch [1] is still unstable. Unit tests can pass or fail in a in a random order. Also, I've disabled some tests to pass CI.</div>
<div><br>
</div>
<div><br>
</div>
<div>Long version:</div>
<div><br>
</div>
<div>While I was working on patch "Move drivers unit tests to unit.volume.drivers directory" [1] I've found a lot of issues with our unit tests :(. Not all of them are already fixed, so that patch is still in progress</div>
<div><br>
</div>
<div>What did I found and what should we have to fix:</div>
<div><br>
</div>
<div>1) Execution time [2]. I don't want to argue what it unit tests, but 2-4 seconds per tests should be non-acceptable, IMO.<br>
</div>
<div><br>
</div>
<div>2) Execution order. Seriously, do you know that our tests will fail or hang if execution order will change? Even if one test for diver A failed, some tests for driver B will fail too.</div>
<div><br>
</div>
<div>3) Lack of mock. It's a root cause for #2. We didn't mock sleeps and event loops right. We don't mock RPC call well too [3]. We don't have 'cinder.openstack.common.rpc.impl_fake' module in Cinder tree.</div>
<div><br>
</div>
<div>In some drivers, we use oslo_service.loopingcall.FixedIntervalLoopingCall [4]. We've go ZeroIntervalLoopingCall [5] class in Cinder. Do we use it everywhere or mock FixedIntervalLoopingCall right? I don't think so, I've hacked oslo_service in my env to
rise an exception if interval > 0. 297 tests failed. It means, our tests use sleep. We have to get rid of this. TBH, not only volume drivers unit tests failed. E.g. some API unit tests failed too.</div>
<div><br>
</div>
<div><br>
</div>
<div>4) Due to #3, sometimes unit tests hangs even on master branch with a minor changes.If I stop execution of such tests, usually I see something like [6]. In most of cases I see that following drivers' tests hangs: EMC, Huawei, Dell and RBD.</div>
<div><br>
</div>
<div>It's hard to debug such failures because the lack of tooling for eventlet debugging. Eventlet backdoors and gdb-python helps a bit. Maybe somebody know better solution for it.</div>
<div><br>
</div>
<div>[1] <a href="https://review.openstack.org/#/c/320148/" target="_blank">https://review.openstack.org/#/c/320148/</a></div>
<div>[2] <a href="http://paste.openstack.org/show/539081/" target="_blank">http://paste.openstack.org/show/539081/</a></div>
<div>[3] <a href="https://github.com/openstack/cinder/search?utf8=%E2%9C%93&q=impl_fake" target="_blank">https://github.com/openstack/cinder/search?utf8=%E2%9C%93&q=impl_fake</a></div>
<div>[4] use <a href="https://github.com/openstack/oslo.service/blob/master/oslo_service/loopingcall.py#L162" target="_blank">https://github.com/openstack/oslo.service/blob/master/oslo_service/loopingcall.py#L162</a></div>
<div>[5] <a href="https://github.com/openstack/cinder/blob/cfbb5bde4d9b37c39f6813fe685f987f8a990483/cinder/tests/unit/utils.py#L289" target="_blank">https://github.com/openstack/cinder/blob/cfbb5bde4d9b37c39f6813fe685f987f8a990483/cinder/tests/unit/utils.py#L289</a></div>
<div>[6] <a href="http://paste.openstack.org/show/539090/" target="_blank">http://paste.openstack.org/show/539090/</a></div>
<div><br>
</div>
<div><br clear="all">
<div>
<div>
<div dir="ltr">
<div>Regards,<br>
Ivan Kolodyazhny,<br>
<a href="http://blog.e0ne.info/" target="_blank">http://blog.e0ne.info/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div></div></div>
</div>
</div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>