<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<font face="serif">GlusterFS CI job is still failing with the same
issue.<br>
<br>
I gave couple of "recheck"s on [1], after </font><font
face="serif"><a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/181288/">https://review.openstack.org/#/c/181288/</a> patch got
merged.<br>
<br>
But still GlusterFS CI job is failing with below error [2]:</font><br>
<font face="serif">
<meta http-equiv="content-type" content="text/html;
charset=windows-1252">
<span style="color: rgb(204, 102, 0); font-family: monospace;
font-size: medium; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent: -286px;
text-transform: none; white-space: pre-wrap; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; display:
inline !important; float: none;">ObjectDereferencedError: Can't
emit change event for attribute 'Volume.provider_location' -
parent object of type <Volume> has been garbage collected.</span><br>
<br>
Also I found the same behaviour with NetApp CI also.<br>
<br>
<br>
[1] <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/165424/">https://review.openstack.org/#/c/165424/</a><br>
[2]
<a class="moz-txt-link-freetext" href="http://logs.openstack.org/24/165424/6/check/check-tempest-dsvm-full-glusterfs-nv/f386477/logs/screen-c-vol.txt.gz">http://logs.openstack.org/24/165424/6/check/check-tempest-dsvm-full-glusterfs-nv/f386477/logs/screen-c-vol.txt.gz</a><br>
<br>
<br>
</font>
<div class="moz-cite-prefix">On 05/08/2015 10:21 AM, Joshua Harlow
wrote:<br>
</div>
<blockquote
cite="mid:BLU436-SMTP2421AACE09AAFE37A379B7ED8DE0@phx.gbl"
type="cite">Alright, it was as I had a hunch for, a small bug
found in the new algorithm to make the storage layer
copy-original,mutate-copy,save-copy,update-original (vs
update-original,save-original) more reliable.
<br>
<br>
<a class="moz-txt-link-freetext" href="https://bugs.launchpad.net/taskflow/+bug/1452978">https://bugs.launchpad.net/taskflow/+bug/1452978</a> opened and a one
line fix made @ <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/181288/">https://review.openstack.org/#/c/181288/</a> to stop
trying to copy task results (which was activating logic that must
of caused the reference to drop out of existence and therefore the
issue noted below).
<br>
<br>
Will get that released in 0.10.1 once it flushes through the
pipeline.
<br>
<br>
Thanks alex for helping double check, if others want to check to
that'd be nice, can make sure that's the root cause (overzealous
usage of copy.copy, ha).
<br>
<br>
Overall I'd still *highly* recommend that the following still
happen:
<br>
<br>
>> One way to get around whatever the issue is would be to
change the
<br>
>> drivers to not update the object directly as it is not
needed. But
<br>
>> this should not fail. Perhaps a more proper fix is for
the volume
<br>
>> manager to not pass around sqlalchemy objects.
<br>
<br>
But that can be a later tweak that cinder does; using any taskflow
engine that isn't the greenthreaded/threaded/serial engine will
require results to be serializable, and therefore copyable, so
that those results can go across IPC or MQ/other boundaries.
Sqlalchemy objects won't fit either of these cases (obviously).
<br>
<br>
-Josh
<br>
<br>
Joshua Harlow wrote:
<br>
<blockquote type="cite">Are we sure this is taskflow? I'm
wondering since those errors are more
<br>
from task code (which is in cinder) and the following seems to
be a
<br>
general garbage collection issue (not connected to taskflow?):
<br>
<br>
'Exception during message handling: Can't emit change event for
<br>
attribute 'Volume.provider_location' - parent object of type
<Volume>
<br>
has been garbage collected.'''
<br>
<br>
Or:
<br>
<br>
'''2015-05-07 22:42:51.142 17040 TRACE
oslo_messaging.rpc.dispatcher
<br>
ObjectDereferencedError: Can't emit change event for attribute
<br>
'Volume.provider_location' - parent object of type
<Volume> has been
<br>
garbage collected.'''
<br>
<br>
Alex Meade wrote:
<br>
<blockquote type="cite">So it seems that this will break a
number of drivers, I see that
<br>
glusterfs does the same thing.
<br>
<br>
On Thu, May 7, 2015 at 10:29 PM, Alex Meade
<<a class="moz-txt-link-abbreviated" href="mailto:mr.alex.meade@gmail.com">mr.alex.meade@gmail.com</a>
<br>
<a class="moz-txt-link-rfc2396E" href="mailto:mr.alex.meade@gmail.com"><mailto:mr.alex.meade@gmail.com></a>> wrote:
<br>
<br>
It appears that the release of taskflow 0.10.0 exposed an
issue in
<br>
the NetApp NFS drivers. Something changed that caused the
sqlalchemy
<br>
Volume object to be garbage collected even though it is passed
into
<br>
create_volume()
<br>
<br>
An example error can be found in the c-vol logs here:
<br>
<br>
<a class="moz-txt-link-freetext" href="http://dcf901611175aa43f968-c54047c910227e27e1d6f03bb1796fd7.r95.cf5.rackcdn.com/57/181157/1/check/cinder-cDOT-NFS/0473c54/">http://dcf901611175aa43f968-c54047c910227e27e1d6f03bb1796fd7.r95.cf5.rackcdn.com/57/181157/1/check/cinder-cDOT-NFS/0473c54/</a>
<br>
<br>
<br>
One way to get around whatever the issue is would be to change
the
<br>
drivers to not update the object directly as it is not needed.
But
<br>
this should not fail. Perhaps a more proper fix is for the
volume
<br>
manager to not pass around sqlalchemy objects.
<br>
</blockquote>
<br>
+1
<br>
<br>
<blockquote type="cite">
<br>
Something changed in taskflow, however, and we should just
<br>
understand if that has other impact.
<br>
</blockquote>
<br>
I'd like to understand that also: the only one commit that
touched this
<br>
stuff is <a class="moz-txt-link-freetext" href="https://github.com/openstack/taskflow/commit/227cf52">https://github.com/openstack/taskflow/commit/227cf52</a>
(which
<br>
basically ensured that a storage object copy is modified, then
saved,
<br>
then the local object is updated vs updating the local object,
and then
<br>
saving, which has problems/inconsistencies if the save fails).
<br>
<br>
<blockquote type="cite">
<br>
-Alex
<br>
<br>
<br>
__________________________________________________________________________
<br>
<br>
OpenStack Development Mailing List (not for usage questions)
<br>
Unsubscribe:
<br>
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<br>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
<br>
</blockquote>
<br>
__________________________________________________________________________
<br>
OpenStack Development Mailing List (not for usage questions)
<br>
Unsubscribe:
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<br>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
<br>
</blockquote>
<br>
__________________________________________________________________________
<br>
OpenStack Development Mailing List (not for usage questions)
<br>
Unsubscribe:
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<br>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Warm Regards,
Bharat Kumar Kobagana
Software Engineer
OpenStack Storage – RedHat India
Mobile - +91 9949278005</pre>
</body>
</html>