Open Stack

Alright, it was as I had a hunch for, a small bug found in the new 
algorithm to make the storage layer 
copy-original,mutate-copy,save-copy,update-original (vs 
update-original,save-original) more reliable.

https://bugs.launchpad.net/taskflow/+bug/1452978 opened and a one line 
fix made @ https://review.openstack.org/#/c/181288/ to stop trying to 
copy task results (which was activating logic that must of caused the 
reference to drop out of existence and therefore the issue noted below).

Will get that released in 0.10.1 once it flushes through the pipeline.

Thanks alex for helping double check, if others want to check to that'd 
be nice, can make sure that's the root cause (overzealous usage of 
copy.copy, ha).

Overall I'd still *highly* recommend that the following still happen:

 >> One way to get around whatever the issue is would be to change the
 >> drivers to not update the object directly as it is not needed. But
 >> this should not fail. Perhaps a more proper fix is for the volume
 >> manager to not pass around sqlalchemy objects.

But that can be a later tweak that cinder does; using any taskflow 
engine that isn't the greenthreaded/threaded/serial engine will require 
results to be serializable, and therefore copyable, so that those 
results can go across IPC or MQ/other boundaries. Sqlalchemy objects 
won't fit either of these cases (obviously).

-Josh

Joshua Harlow wrote:
> Are we sure this is taskflow? I'm wondering since those errors are more
> from task code (which is in cinder) and the following seems to be a
> general garbage collection issue (not connected to taskflow?):
>
> 'Exception during message handling: Can't emit change event for
> attribute 'Volume.provider_location' - parent object of type <Volume>
> has been garbage collected.'''
>
> Or:
>
> '''2015-05-07 22:42:51.142 17040 TRACE oslo_messaging.rpc.dispatcher
> ObjectDereferencedError: Can't emit change event for attribute
> 'Volume.provider_location' - parent object of type <Volume> has been
> garbage collected.'''
>
> Alex Meade wrote:
>> So it seems that this will break a number of drivers, I see that
>> glusterfs does the same thing.
>>
>> On Thu, May 7, 2015 at 10:29 PM, Alex Meade <mr.alex.meade at gmail.com
>> <mailto:mr.alex.meade at gmail.com>> wrote:
>>
>> It appears that the release of taskflow 0.10.0 exposed an issue in
>> the NetApp NFS drivers. Something changed that caused the sqlalchemy
>> Volume object to be garbage collected even though it is passed into
>> create_volume()
>>
>> An example error can be found in the c-vol logs here:
>>
>> http://dcf901611175aa43f968-c54047c910227e27e1d6f03bb1796fd7.r95.cf5.rackcdn.com/57/181157/1/check/cinder-cDOT-NFS/0473c54/
>>
>>
>> One way to get around whatever the issue is would be to change the
>> drivers to not update the object directly as it is not needed. But
>> this should not fail. Perhaps a more proper fix is for the volume
>> manager to not pass around sqlalchemy objects.
>
> +1
>
>>
>> Something changed in taskflow, however, and we should just
>> understand if that has other impact.
>
> I'd like to understand that also: the only one commit that touched this
> stuff is https://github.com/openstack/taskflow/commit/227cf52 (which
> basically ensured that a storage object copy is modified, then saved,
> then the local object is updated vs updating the local object, and then
> saving, which has problems/inconsistencies if the save fails).
>
>>
>> -Alex
>>
>>
>> __________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev