<div dir="ltr">Eric, you're right.<div><br></div>I've disabled all such tests using '@unittest.skip("Skip until bug #1578986 is fixed")' decorator in my patch:<div><div>$ grep -r '1578986' cinder/tests/unit/ | grep -v 'pyc' | wc -l</div><div>37</div></div><div><br></div><div>Next step is to fix them.</div><div><br><div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">[1] </span><a href="https://review.openstack.org/#/c/320148/" style="font-size:12.8px" target="_blank">https://review.openstack.org/#/c/320148/</a></div></div></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Regards,<br>Ivan Kolodyazhny,<br><a href="http://blog.e0ne.info/" target="_blank">http://blog.e0ne.info/</a></div></div></div></div>
<br><div class="gmail_quote">On Fri, Jul 22, 2016 at 4:13 PM, Eric Harney <span dir="ltr"><<a href="mailto:eharney@redhat.com" target="_blank">eharney@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 07/21/2016 05:26 PM, Knight, Clinton wrote:<br>
> Nate, you have to press Ctrl-C to see the in-progress test, that’s why you don’t<br>
> see it in the logs. The bug report shows this and points to the patch where it<br>
> appeared to begin. <a href="https://bugs.launchpad.net/cinder/+bug/1578986" rel="noreferrer" target="_blank">https://bugs.launchpad.net/cinder/+bug/1578986</a><br>
><br>
> Clinton<br>
><br>
<br>
</span>I think this only gives a backtrace of the test runner and not the test.<br>
<br>
I attached gdb when this hang occured and see this. Looks like we still<br>
have a thread running the oslo.messaging fake driver.<br>
<br>
<a href="http://paste.openstack.org/raw/539769/" rel="noreferrer" target="_blank">http://paste.openstack.org/raw/539769/</a><br>
<br>
(Linked in the bug report as well.)<br>
<br>
> *From: *"Potter, Nathaniel" <<a href="mailto:nathaniel.potter@intel.com">nathaniel.potter@intel.com</a>><br>
> *Reply-To: *"OpenStack Development Mailing List (not for usage questions)"<br>
> <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
> *Date: *Thursday, July 21, 2016 at 7:17 PM<br>
> *To: *"OpenStack Development Mailing List (not for usage questions)"<br>
> <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
> *Subject: *Re: [openstack-dev] [cinder] Volume Drivers unit tests<br>
<span class="">><br>
> Hi all,<br>
><br>
> I’m not totally sure that this is the same issue, but lately I’ve seen the gate<br>
> tests fail while hanging at this point [1], but they say ‘ok’ rather than<br>
> ‘inprogress’. Has anyone else come across this? It only happens sometimes, and a<br>
> recheck can get past it. The full log is here [2].<br>
><br>
> [1] <a href="http://paste.openstack.org/show/539314/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/539314/</a><br>
><br>
> [2]<br>
> <a href="http://logs.openstack.org/90/341090/6/check/gate-cinder-python34-db/ea65de5/console.html" rel="noreferrer" target="_blank">http://logs.openstack.org/90/341090/6/check/gate-cinder-python34-db/ea65de5/console.html</a><br>
><br>
> Thanks,<br>
><br>
> Nate<br>
><br>
</span>> *From:*yang, xing [mailto:<a href="mailto:xing.yang@emc.com">xing.yang@emc.com</a>]<br>
> *Sent:* Thursday, July 21, 2016 3:17 PM<br>
> *To:* OpenStack Development Mailing List (not for usage questions)<br>
> <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
> *Subject:* Re: [openstack-dev] [cinder] Volume Drivers unit tests<br>
<span class="">><br>
> Hi Ivan,<br>
><br>
> Do you have any logs for the VMAX driver? We'll take a look.<br>
><br>
> Thanks,<br>
><br>
> Xing<br>
><br>
</span>> --------------------------------------------------------------------------------<br>
><br>
> *From:*Ivan Kolodyazhny [<a href="mailto:e0ne@e0ne.info">e0ne@e0ne.info</a>]<br>
> *Sent:* Thursday, July 21, 2016 4:44 PM<br>
> *To:* OpenStack Development Mailing List (not for usage questions)<br>
> *Subject:* Re: [openstack-dev] [cinder] Volume Drivers unit tests<br>
<span class="">><br>
> Thank you Xing,<br>
><br>
> The issue is related both to VNX and VMAX EMC drivers<br>
><br>
><br>
> Regards,<br>
> Ivan Kolodyazhny,<br>
> <a href="http://blog.e0ne.info/" rel="noreferrer" target="_blank">http://blog.e0ne.info/</a><br>
><br>
> On Thu, Jul 21, 2016 at 11:00 PM, yang, xing <<a href="mailto:xing.yang@emc.com">xing.yang@emc.com</a><br>
</span><span class="">> <mailto:<a href="mailto:xing.yang@emc.com">xing.yang@emc.com</a>>> wrote:<br>
><br>
> Hi Ivan,<br>
><br>
> Thanks for sending this out. Regarding the issue in the EMC VNX driver unit<br>
> tests, it is tracked by this bug<br>
> <a href="https://bugs.launchpad.net/cinder/+bug/1578986" rel="noreferrer" target="_blank">https://bugs.launchpad.net/cinder/+bug/1578986</a>. The driver was recently<br>
> refactored so this is probably a new issue introduced by the refactor.<br>
> We are investigating this issue.<br>
><br>
> Thanks,<br>
><br>
> Xing<br>
><br>
</span>> --------------------------------------------------------------------------------<br>
><br>
> *From:*Ivan Kolodyazhny [<a href="mailto:e0ne@e0ne.info">e0ne@e0ne.info</a> <mailto:<a href="mailto:e0ne@e0ne.info">e0ne@e0ne.info</a>>]<br>
> *Sent:* Thursday, July 21, 2016 1:02 PM<br>
> *To:* OpenStack Development Mailing List<br>
> *Subject:* [openstack-dev] [cinder] Volume Drivers unit tests<br>
<div><div class="h5">><br>
> Hi team,<br>
><br>
> First of all, I would like to apologize, if my mail is be too emotional. I<br>
> spent too much of time to fix it and failed.<br>
><br>
> TL;DR;<br>
><br>
> What I want to say is: "Let's spend some time to make our tests better and<br>
> fix all issues". Patch [1] is still unstable. Unit tests can pass or fail in<br>
> a in a random order. Also, I've disabled some tests to pass CI.<br>
><br>
> Long version:<br>
><br>
> While I was working on patch "Move drivers unit tests to unit.volume.drivers<br>
> directory" [1] I've found a lot of issues with our unit tests :(. Not all of<br>
> them are already fixed, so that patch is still in progress<br>
><br>
> What did I found and what should we have to fix:<br>
><br>
> 1) Execution time [2]. I don't want to argue what it unit tests, but 2-4<br>
> seconds per tests should be non-acceptable, IMO.<br>
><br>
> 2) Execution order. Seriously, do you know that our tests will fail or hang<br>
> if execution order will change? Even if one test for diver A failed, some<br>
> tests for driver B will fail too.<br>
><br>
> 3) Lack of mock. It's a root cause for #2. We didn't mock sleeps and event<br>
> loops right. We don't mock RPC call well too [3]. We don't<br>
> have 'cinder.openstack.common.rpc.impl_fake' module in Cinder tree.<br>
><br>
> In some drivers, we use oslo_service.loopingcall.FixedIntervalLoopingCall<br>
> [4]. We've go ZeroIntervalLoopingCall [5] class in Cinder. Do we use it<br>
> everywhere or mock FixedIntervalLoopingCall right? I don't think so, I've<br>
> hacked oslo_service in my env to rise an exception if interval > 0. 297<br>
> tests failed. It means, our tests use sleep. We have to get rid of this.<br>
> TBH, not only volume drivers unit tests failed. E.g. some API unit tests<br>
> failed too.<br>
><br>
> 4) Due to #3, sometimes unit tests hangs even on master branch with a minor<br>
> changes.If I stop execution of such tests, usually I see something like [6].<br>
> In most of cases I see that following drivers' tests hangs: EMC, Huawei,<br>
> Dell and RBD.<br>
><br>
> It's hard to debug such failures because the lack of tooling for eventlet<br>
> debugging. Eventlet backdoors and gdb-python helps a bit. Maybe somebody<br>
> know better solution for it.<br>
><br>
> [1] <a href="https://review.openstack.org/#/c/320148/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/320148/</a><br>
><br>
> [2] <a href="http://paste.openstack.org/show/539081/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/539081/</a><br>
><br>
> [3] <a href="https://github.com/openstack/cinder/search?utf8=%E2%9C%93&q=impl_fake" rel="noreferrer" target="_blank">https://github.com/openstack/cinder/search?utf8=%E2%9C%93&q=impl_fake</a><br>
><br>
> [4] use<br>
> <a href="https://github.com/openstack/oslo.service/blob/master/oslo_service/loopingcall.py#L162" rel="noreferrer" target="_blank">https://github.com/openstack/oslo.service/blob/master/oslo_service/loopingcall.py#L162</a><br>
><br>
> [5]<br>
> <a href="https://github.com/openstack/cinder/blob/cfbb5bde4d9b37c39f6813fe685f987f8a990483/cinder/tests/unit/utils.py#L289" rel="noreferrer" target="_blank">https://github.com/openstack/cinder/blob/cfbb5bde4d9b37c39f6813fe685f987f8a990483/cinder/tests/unit/utils.py#L289</a><br>
><br>
> [6] <a href="http://paste.openstack.org/show/539090/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/539090/</a><br>
><br>
><br>
> Regards,<br>
> Ivan Kolodyazhny,<br>
> <a href="http://blog.e0ne.info/" rel="noreferrer" target="_blank">http://blog.e0ne.info/</a><br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
</div></div>> <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<div class="HOEnZb"><div class="h5">><br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>