Open Stack

Eric, you're right.

I've disabled all such tests using '@unittest.skip("Skip until bug #1578986
is fixed")' decorator in my patch:
$ grep -r '1578986' cinder/tests/unit/  | grep -v 'pyc' | wc -l
37

Next step is to fix them.


[1] https://review.openstack.org/#/c/320148/

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Fri, Jul 22, 2016 at 4:13 PM, Eric Harney <eharney at redhat.com> wrote:

> On 07/21/2016 05:26 PM, Knight, Clinton wrote:
> > Nate, you have to press Ctrl-C to see the in-progress test, that’s why
> you don’t
> > see it in the logs.  The bug report shows this and points to the patch
> where it
> > appeared to begin. https://bugs.launchpad.net/cinder/+bug/1578986
> >
> > Clinton
> >
>
> I think this only gives a backtrace of the test runner and not the test.
>
> I attached gdb when this hang occured and see this.  Looks like we still
> have a thread running the oslo.messaging fake driver.
>
> http://paste.openstack.org/raw/539769/
>
> (Linked in the bug report as well.)
>
> > *From: *"Potter, Nathaniel" <nathaniel.potter at intel.com>
> > *Reply-To: *"OpenStack Development Mailing List (not for usage
> questions)"
> > <openstack-dev at lists.openstack.org>
> > *Date: *Thursday, July 21, 2016 at 7:17 PM
> > *To: *"OpenStack Development Mailing List (not for usage questions)"
> > <openstack-dev at lists.openstack.org>
> > *Subject: *Re: [openstack-dev] [cinder] Volume Drivers unit tests
> >
> > Hi all,
> >
> > I’m not totally sure that this is the same issue, but lately I’ve seen
> the gate
> > tests fail while hanging at this point [1], but they say ‘ok’ rather than
> > ‘inprogress’. Has anyone else come across this? It only happens
> sometimes, and a
> > recheck can get past it. The full log is here [2].
> >
> > [1] http://paste.openstack.org/show/539314/
> >
> > [2]
> >
> http://logs.openstack.org/90/341090/6/check/gate-cinder-python34-db/ea65de5/console.html
> >
> > Thanks,
> >
> > Nate
> >
> > *From:*yang, xing [mailto:xing.yang at emc.com]
> > *Sent:* Thursday, July 21, 2016 3:17 PM
> > *To:* OpenStack Development Mailing List (not for usage questions)
> > <openstack-dev at lists.openstack.org>
> > *Subject:* Re: [openstack-dev] [cinder] Volume Drivers unit tests
> >
> > Hi Ivan,
> >
> > Do you have any logs for the VMAX driver?  We'll take a look.
> >
> > Thanks,
> >
> > Xing
> >
> >
> --------------------------------------------------------------------------------
> >
> > *From:*Ivan Kolodyazhny [e0ne at e0ne.info]
> > *Sent:* Thursday, July 21, 2016 4:44 PM
> > *To:* OpenStack Development Mailing List (not for usage questions)
> > *Subject:* Re: [openstack-dev] [cinder] Volume Drivers unit tests
> >
> > Thank you Xing,
> >
> > The issue is related both to VNX and VMAX EMC drivers
> >
> >
> > Regards,
> > Ivan Kolodyazhny,
> > http://blog.e0ne.info/
> >
> > On Thu, Jul 21, 2016 at 11:00 PM, yang, xing <xing.yang at emc.com
> > <mailto:xing.yang at emc.com>> wrote:
> >
> >     Hi Ivan,
> >
> >     Thanks for sending this out.  Regarding the issue in the EMC VNX
> driver unit
> >     tests, it is tracked by this bug
> >     https://bugs.launchpad.net/cinder/+bug/1578986. The driver was
> recently
> >     refactored so this is probably a new issue introduced by the
> refactor.
> >     We are investigating this issue.
> >
> >     Thanks,
> >
> >     Xing
> >
> >
>  --------------------------------------------------------------------------------
> >
> >     *From:*Ivan Kolodyazhny [e0ne at e0ne.info <mailto:e0ne at e0ne.info>]
> >     *Sent:* Thursday, July 21, 2016 1:02 PM
> >     *To:* OpenStack Development Mailing List
> >     *Subject:* [openstack-dev] [cinder] Volume Drivers unit tests
> >
> >     Hi team,
> >
> >     First of all, I would like to apologize, if my mail is be too
> emotional. I
> >     spent too much of time to fix it and failed.
> >
> >     TL;DR;
> >
> >     What I want to say is: "Let's spend some time to make our tests
> better and
> >     fix all issues". Patch [1] is still unstable. Unit tests can pass or
> fail in
> >     a in a random order. Also, I've disabled some tests to pass CI.
> >
> >     Long version:
> >
> >     While I was working on patch "Move drivers unit tests to
> unit.volume.drivers
> >     directory" [1] I've found a lot of issues with our unit tests :(.
> Not all of
> >     them are already fixed, so that patch is still in progress
> >
> >     What did I found and what should we have to fix:
> >
> >     1) Execution time [2]. I don't want to argue what it unit tests, but
> 2-4
> >     seconds per tests should be non-acceptable, IMO.
> >
> >     2) Execution order. Seriously, do you know that our tests will fail
> or hang
> >     if execution order will change? Even if one test for diver A failed,
> some
> >     tests for driver B will fail too.
> >
> >     3) Lack of mock. It's a root cause for #2. We didn't mock sleeps and
> event
> >     loops right. We don't mock RPC call well too [3]. We don't
> >     have 'cinder.openstack.common.rpc.impl_fake' module in Cinder tree.
> >
> >     In some drivers, we use
> oslo_service.loopingcall.FixedIntervalLoopingCall
> >     [4]. We've go ZeroIntervalLoopingCall [5] class in Cinder. Do we use
> it
> >     everywhere or mock FixedIntervalLoopingCall right? I don't think so,
> I've
> >     hacked oslo_service in my env to rise an exception if interval > 0.
> 297
> >     tests failed. It means, our tests use sleep. We have to get rid of
> this.
> >     TBH, not only volume drivers unit tests failed. E.g. some API unit
> tests
> >     failed too.
> >
> >     4) Due to #3, sometimes unit tests hangs even on master branch with
> a minor
> >     changes.If I stop execution of such tests, usually I see something
> like [6].
> >     In most of cases I see that following drivers' tests hangs: EMC,
> Huawei,
> >     Dell and RBD.
> >
> >     It's hard to debug such failures because the lack of tooling for
> eventlet
> >     debugging. Eventlet backdoors and gdb-python helps a bit. Maybe
> somebody
> >     know better solution for it.
> >
> >     [1] https://review.openstack.org/#/c/320148/
> >
> >     [2] http://paste.openstack.org/show/539081/
> >
> >     [3]
> https://github.com/openstack/cinder/search?utf8=%E2%9C%93&q=impl_fake
> >
> >     [4] use
> >
> https://github.com/openstack/oslo.service/blob/master/oslo_service/loopingcall.py#L162
> >
> >     [5]
> >
> https://github.com/openstack/cinder/blob/cfbb5bde4d9b37c39f6813fe685f987f8a990483/cinder/tests/unit/utils.py#L289
> >
> >     [6] http://paste.openstack.org/show/539090/
> >
> >
> >     Regards,
> >     Ivan Kolodyazhny,
> >     http://blog.e0ne.info/
> >
> >
> >
>  __________________________________________________________________________
> >     OpenStack Development Mailing List (not for usage questions)
> >     Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >     <
> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> >     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160722/674c50dd/attachment.html>