Open Stack

On Mon, Feb 3, 2014 at 1:53 PM, Joshua Harlow <harlowja at yahoo-inc.com> wrote:
> Hi all,
>
> After talking with john g. about taskflow in cinder and seeing more and
> more reviews showing up I wanted to start a thread to gather all our
> lessons learned and how we can improve a little before continuing to add
> too many more refactoring and more reviews (making sure everyone is
> understands the larger goal and larger picture of switching pieces of
> cinder - piece by piece - to taskflow).
>
> Just to catch everyone up.
>
> Taskflow started integrating with cinder in havana and there has been some
> continued work around these changes:
>
> - https://review.openstack.org/#/c/58724/
> - https://review.openstack.org/#/c/66283/
> - https://review.openstack.org/#/c/62671/
>
> There has also been a few other pieces of work going in (forgive me if I
> missed any...):
>
> - https://review.openstack.org/#/c/64469/
> - https://review.openstack.org/#/c/69329/
> - https://review.openstack.org/#/c/64026/
>
> I think now would be a good time (and seems like a good idea) to create
> the discussion to learn how people are using taskflow, common patterns
> people like, don't like, common refactoring idioms that are occurring and
> most importantly to make sure that we refactor with a purpose and not just
> refactor for refactoring sake (which can be harmful if not done
> correctly). So to get a kind of forward and unified momentum behind
> further adjustments I'd just like to make sure we are all aligned and
> understood on the benefits and yes even the drawbacks that these
> refactorings bring.
>
> So here is my little list of benefits:
>
> - Objects that do just one thing (a common pattern I am seeing is
> determining what the one thing is, without making it to granular that its
> hard to read).
> - Combining these objects together in a well-defined way (once again it
> has to be carefully done to not create to much granularness).
> - Ability to test these tasks and flows via mocking (something that is
> harder when its not split up like this).
> - Features that aren't currently used such as state-persistence (but will
> help cinder become more crash-resistant in the future).
>   - This one will itself need to be understood before doing [I started
> etherpad @ https://etherpad.openstack.org/p/cinder-taskflow-persistence
> for this].
>
> List of drawbacks (or potential drawbacks):
>
> - Having a understanding of what taskflow is doing adds on a new layer of
> things to know (hopefully docs help in this area, that was there goal).
> - Selecting to granular of a task or flow; makes it harder to
> follow/understand the task/flow logic.
> - Focuses on the long-term (not necessarily short-term) state-management
> concerns (can't refactor rome in a day).
> - Taskflow is being developed at the same time cinder is.
>
> I'd be very interested in hearing about others experiences and to make
> sure that we discuss the changes (in a well documented and agreed on
> approach) before jumping to much into the 'deep end' with a large amount
> of refactoring (aka, refactoring with a purpose). Let's make this thread
> as useful as we can and try to see how we can unify all these refactorings
> behind a common (and documented & agreed-on) purpose.
>
> A thought, for the reviews above, I think it would be very useful to
> etherpad/writeup more in the blueprint what the 'refactoring with a
> purpose' is so that its more known to future readers (and for active
> reviewers), hopefully this email can start to help clarify that purpose so
> that things proceed as smoothly as possible.
>
> -Josh
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Thanks for putting this together Josh, I just wanted to add a couple
of things from my own perspective.

The end-goals of taskflow (specifically persistence and better state
managment) are the motivating factors for going this route.  We've
made a first step with create_volume however we haven't advanced it
enough to realize the benefits that we set out to gain by this in the
first place.  I still think it's the right direction and IMO we should
keep on the path, however there are a number of things that I've
noticed that make me lean towards refraining from moving other API
calls to taskflow right now.

1. Currently taskflow is pretty much a functional equivalent
replacement of what was in the volume manager.  We're not really
gaining that much from it (yet).

2. taskflow adds quite a bit of code and indirection that currently
IMHO adds a bit of complexity and difficulty in trouble-shooting (I
think we're fixing this up and it will continue to get better, I also
think this is normal for introduction of new implementations, no
criticism intended).

3. Our unit testing / mock infrastructure is broken right now for
items that use taskflow.  Particularly cinder.test.test_volume can not
be run independently until we fix the taskflow fakes and mock objects.
 I def don't want anything else taskflow related merged until this
problem is addressed.

4. We really haven't come up with solutions to the problems we set out
to solve in the first place with our first implementation of taskflow
(state management and persistence).  Until we have a pattern for
solving this I think we should refrain from implementing it in other
calls.  A number of people volunteered to work on this at the summit
in Hong Kong and have stated that they "have code" however that code
or those patches haven't materialized so I think we need to regroup
and get this work moving again.

Anyway, I'd like to stabilize things for the create_volume
implementation that we have and have a clear well defined pattern that
solves problems before we go crazy refactoring every API call to use
taskflow and assume all of the potential risk that goes along with it.

Thanks,
John