<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 12px; font-family: Arial, sans-serif;">
<div>
<div>
<div>
<div>The problem is in the version of taskflow that is downloaded from pypi by devstack. You will need to wait until a new version >0.10.0 is available [1]</div>
<div><br>
</div>
<div>[1] <a href="https://pypi.python.org/pypi/taskflow/">https://pypi.python.org/pypi/taskflow/</a></div>
</div>
<div><br>
</div>
<div>
<div style="font-family: Arial, Helvetica; font-weight: bold;">Andrew Kerr</div>
<div style="font-family: Arial, Helvetica;">OpenStack QA</div>
<div style="font-family: Arial, Helvetica;">Cloud Solutions Group</div>
<div style="font-family: Arial, Helvetica; font-weight: bold;">NetApp</div>
<div style="font-family: Arial, Helvetica;"></div>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Bharat Kumar <<a href="mailto:bharat.kobagana@redhat.com">bharat.kobagana@redhat.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Friday, May 8, 2015 at 7:37 AM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [cinder] Taskflow 0.10.0 incompatible with NetApp NFS drivers<br>
</div>
<div><br>
</div>
<div>
<div bgcolor="#FFFFFF" text="#000000"><font face="serif">GlusterFS CI job is still failing with the same issue.<br>
<br>
I gave couple of "recheck"s on [1], after </font><font face="serif"><a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/181288/">https://review.openstack.org/#/c/181288/</a> patch got merged.<br>
<br>
But still GlusterFS CI job is failing with below error [2]:</font><br>
<font face="serif"><span style="color: rgb(204, 102, 0); font-family: monospace;
font-size: medium; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent: -286px;
text-transform: none; white-space: pre-wrap; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; display:
inline !important; float: none;">ObjectDereferencedError:
Can't emit change event for attribute 'Volume.provider_location' - parent object of type <Volume> has been garbage collected.</span><br>
<br>
Also I found the same behaviour with NetApp CI also.<br>
<br>
<br>
[1] <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/165424/">
https://review.openstack.org/#/c/165424/</a><br>
[2] <a class="moz-txt-link-freetext" href="http://logs.openstack.org/24/165424/6/check/check-tempest-dsvm-full-glusterfs-nv/f386477/logs/screen-c-vol.txt.gz">
http://logs.openstack.org/24/165424/6/check/check-tempest-dsvm-full-glusterfs-nv/f386477/logs/screen-c-vol.txt.gz</a><br>
<br>
<br>
</font>
<div class="moz-cite-prefix">On 05/08/2015 10:21 AM, Joshua Harlow wrote:<br>
</div>
<blockquote cite="mid:BLU436-SMTP2421AACE09AAFE37A379B7ED8DE0@phx.gbl" type="cite">
Alright, it was as I had a hunch for, a small bug found in the new algorithm to make the storage layer copy-original,mutate-copy,save-copy,update-original (vs update-original,save-original) more reliable.
<br>
<br>
<a class="moz-txt-link-freetext" href="https://bugs.launchpad.net/taskflow/+bug/1452978">https://bugs.launchpad.net/taskflow/+bug/1452978</a> opened and a one line fix made @
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/181288/">
https://review.openstack.org/#/c/181288/</a> to stop trying to copy task results (which was activating logic that must of caused the reference to drop out of existence and therefore the issue noted below).
<br>
<br>
Will get that released in 0.10.1 once it flushes through the pipeline. <br>
<br>
Thanks alex for helping double check, if others want to check to that'd be nice, can make sure that's the root cause (overzealous usage of copy.copy, ha).
<br>
<br>
Overall I'd still *highly* recommend that the following still happen: <br>
<br>
>> One way to get around whatever the issue is would be to change the <br>
>> drivers to not update the object directly as it is not needed. But <br>
>> this should not fail. Perhaps a more proper fix is for the volume <br>
>> manager to not pass around sqlalchemy objects. <br>
<br>
But that can be a later tweak that cinder does; using any taskflow engine that isn't the greenthreaded/threaded/serial engine will require results to be serializable, and therefore copyable, so that those results can go across IPC or MQ/other boundaries. Sqlalchemy
objects won't fit either of these cases (obviously). <br>
<br>
-Josh <br>
<br>
Joshua Harlow wrote: <br>
<blockquote type="cite">Are we sure this is taskflow? I'm wondering since those errors are more
<br>
from task code (which is in cinder) and the following seems to be a <br>
general garbage collection issue (not connected to taskflow?): <br>
<br>
'Exception during message handling: Can't emit change event for <br>
attribute 'Volume.provider_location' - parent object of type <Volume> <br>
has been garbage collected.''' <br>
<br>
Or: <br>
<br>
'''2015-05-07 22:42:51.142 17040 TRACE oslo_messaging.rpc.dispatcher <br>
ObjectDereferencedError: Can't emit change event for attribute <br>
'Volume.provider_location' - parent object of type <Volume> has been <br>
garbage collected.''' <br>
<br>
Alex Meade wrote: <br>
<blockquote type="cite">So it seems that this will break a number of drivers, I see that
<br>
glusterfs does the same thing. <br>
<br>
On Thu, May 7, 2015 at 10:29 PM, Alex Meade <<a class="moz-txt-link-abbreviated" href="mailto:mr.alex.meade@gmail.com">mr.alex.meade@gmail.com</a><br>
<a class="moz-txt-link-rfc2396E" href="mailto:mr.alex.meade@gmail.com"><mailto:mr.alex.meade@gmail.com></a>> wrote:
<br>
<br>
It appears that the release of taskflow 0.10.0 exposed an issue in <br>
the NetApp NFS drivers. Something changed that caused the sqlalchemy <br>
Volume object to be garbage collected even though it is passed into <br>
create_volume() <br>
<br>
An example error can be found in the c-vol logs here: <br>
<br>
<a class="moz-txt-link-freetext" href="http://dcf901611175aa43f968-c54047c910227e27e1d6f03bb1796fd7.r95.cf5.rackcdn.com/57/181157/1/check/cinder-cDOT-NFS/0473c54/">http://dcf901611175aa43f968-c54047c910227e27e1d6f03bb1796fd7.r95.cf5.rackcdn.com/57/181157/1/check/cinder-cDOT-NFS/0473c54/</a><br>
<br>
<br>
One way to get around whatever the issue is would be to change the <br>
drivers to not update the object directly as it is not needed. But <br>
this should not fail. Perhaps a more proper fix is for the volume <br>
manager to not pass around sqlalchemy objects. <br>
</blockquote>
<br>
+1 <br>
<br>
<blockquote type="cite"><br>
Something changed in taskflow, however, and we should just <br>
understand if that has other impact. <br>
</blockquote>
<br>
I'd like to understand that also: the only one commit that touched this <br>
stuff is <a class="moz-txt-link-freetext" href="https://github.com/openstack/taskflow/commit/227cf52">
https://github.com/openstack/taskflow/commit/227cf52</a> (which <br>
basically ensured that a storage object copy is modified, then saved, <br>
then the local object is updated vs updating the local object, and then <br>
saving, which has problems/inconsistencies if the save fails). <br>
<br>
<blockquote type="cite"><br>
-Alex <br>
<br>
<br>
__________________________________________________________________________ <br>
<br>
OpenStack Development Mailing List (not for usage questions) <br>
Unsubscribe: <br>
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
<br>
__________________________________________________________________________ <br>
OpenStack Development Mailing List (not for usage questions) <br>
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a> <br>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
<br>
__________________________________________________________________________ <br>
OpenStack Development Mailing List (not for usage questions) <br>
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a> <br>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Warm Regards,
Bharat Kumar Kobagana
Software Engineer
OpenStack Storage – RedHat India
Mobile - +91 9949278005</pre>
</div>
</div>
</span>
</body>
</html>