Open Stack

On 07/21/2016 05:26 PM, Knight, Clinton wrote:
> Nate, you have to press Ctrl-C to see the in-progress test, that’s why you don’t 
> see it in the logs.  The bug report shows this and points to the patch where it 
> appeared to begin. https://bugs.launchpad.net/cinder/+bug/1578986
> 
> Clinton
> 

I think this only gives a backtrace of the test runner and not the test.

I attached gdb when this hang occured and see this.  Looks like we still
have a thread running the oslo.messaging fake driver.

http://paste.openstack.org/raw/539769/

(Linked in the bug report as well.)

> *From: *"Potter, Nathaniel" <nathaniel.potter at intel.com>
> *Reply-To: *"OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev at lists.openstack.org>
> *Date: *Thursday, July 21, 2016 at 7:17 PM
> *To: *"OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev at lists.openstack.org>
> *Subject: *Re: [openstack-dev] [cinder] Volume Drivers unit tests
> 
> Hi all,
> 
> I’m not totally sure that this is the same issue, but lately I’ve seen the gate 
> tests fail while hanging at this point [1], but they say ‘ok’ rather than 
> ‘inprogress’. Has anyone else come across this? It only happens sometimes, and a 
> recheck can get past it. The full log is here [2].
> 
> [1] http://paste.openstack.org/show/539314/
> 
> [2] 
> http://logs.openstack.org/90/341090/6/check/gate-cinder-python34-db/ea65de5/console.html
> 
> Thanks,
> 
> Nate
> 
> *From:*yang, xing [mailto:xing.yang at emc.com]
> *Sent:* Thursday, July 21, 2016 3:17 PM
> *To:* OpenStack Development Mailing List (not for usage questions) 
> <openstack-dev at lists.openstack.org>
> *Subject:* Re: [openstack-dev] [cinder] Volume Drivers unit tests
> 
> Hi Ivan,
> 
> Do you have any logs for the VMAX driver?  We'll take a look.
> 
> Thanks,
> 
> Xing
> 
> --------------------------------------------------------------------------------
> 
> *From:*Ivan Kolodyazhny [e0ne at e0ne.info]
> *Sent:* Thursday, July 21, 2016 4:44 PM
> *To:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [cinder] Volume Drivers unit tests
> 
> Thank you Xing,
> 
> The issue is related both to VNX and VMAX EMC drivers
> 
> 
> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/
> 
> On Thu, Jul 21, 2016 at 11:00 PM, yang, xing <xing.yang at emc.com 
> <mailto:xing.yang at emc.com>> wrote:
> 
>     Hi Ivan,
> 
>     Thanks for sending this out.  Regarding the issue in the EMC VNX driver unit
>     tests, it is tracked by this bug
>     https://bugs.launchpad.net/cinder/+bug/1578986. The driver was recently
>     refactored so this is probably a new issue introduced by the refactor. 
>     We are investigating this issue.
> 
>     Thanks,
> 
>     Xing
> 
>     --------------------------------------------------------------------------------
> 
>     *From:*Ivan Kolodyazhny [e0ne at e0ne.info <mailto:e0ne at e0ne.info>]
>     *Sent:* Thursday, July 21, 2016 1:02 PM
>     *To:* OpenStack Development Mailing List
>     *Subject:* [openstack-dev] [cinder] Volume Drivers unit tests
> 
>     Hi team,
> 
>     First of all, I would like to apologize, if my mail is be too emotional. I
>     spent too much of time to fix it and failed.
> 
>     TL;DR;
> 
>     What I want to say is: "Let's spend some time to make our tests better and
>     fix all issues". Patch [1] is still unstable. Unit tests can pass or fail in
>     a in a random order. Also, I've disabled some tests to pass CI.
> 
>     Long version:
> 
>     While I was working on patch "Move drivers unit tests to unit.volume.drivers
>     directory" [1] I've found a lot of issues with our unit tests :(. Not all of
>     them are already fixed, so that patch is still in progress
> 
>     What did I found and what should we have to fix:
> 
>     1) Execution time [2]. I don't want to argue what it unit tests, but 2-4
>     seconds per tests should be non-acceptable, IMO.
> 
>     2) Execution order. Seriously, do you know that our tests will fail or hang
>     if execution order will change? Even if one test for diver A failed, some
>     tests for driver B will fail too.
> 
>     3) Lack of mock. It's a root cause for #2. We didn't mock sleeps and event
>     loops right. We don't mock RPC call well too [3]. We don't
>     have 'cinder.openstack.common.rpc.impl_fake' module in Cinder tree.
> 
>     In some drivers, we use oslo_service.loopingcall.FixedIntervalLoopingCall
>     [4]. We've go ZeroIntervalLoopingCall [5] class in Cinder. Do we use it
>     everywhere or mock FixedIntervalLoopingCall right? I don't think so, I've
>     hacked oslo_service in my env to rise an exception if interval > 0. 297
>     tests failed. It means, our tests use sleep. We have to get rid of this.
>     TBH, not only volume drivers unit tests failed. E.g. some API unit tests
>     failed too.
> 
>     4) Due to #3, sometimes unit tests hangs even on master branch with a minor
>     changes.If I stop execution of such tests, usually I see something like [6].
>     In most of cases I see that following drivers' tests hangs: EMC, Huawei,
>     Dell and RBD.
> 
>     It's hard to debug such failures because the lack of tooling for eventlet
>     debugging. Eventlet backdoors and gdb-python helps a bit. Maybe somebody
>     know better solution for it.
> 
>     [1] https://review.openstack.org/#/c/320148/
> 
>     [2] http://paste.openstack.org/show/539081/
> 
>     [3] https://github.com/openstack/cinder/search?utf8=%E2%9C%93&q=impl_fake
> 
>     [4] use
>     https://github.com/openstack/oslo.service/blob/master/oslo_service/loopingcall.py#L162
> 
>     [5]
>     https://github.com/openstack/cinder/blob/cfbb5bde4d9b37c39f6813fe685f987f8a990483/cinder/tests/unit/utils.py#L289
> 
>     [6] http://paste.openstack.org/show/539090/
> 
> 
>     Regards,
>     Ivan Kolodyazhny,
>     http://blog.e0ne.info/
> 
> 
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>