Open Stack

Inline responses.

From: Renat Akhmerov <rakhmerov at mirantis.com<mailto:rakhmerov at mirantis.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Tuesday, April 1, 2014 at 3:43 AM
To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Subject: Re: [openstack-dev] [Mistral][TaskFlow] Long running actions


On 25 Mar 2014, at 01:51, Joshua Harlow <harlowja at yahoo-inc.com<mailto:harlowja at yahoo-inc.com>> wrote:

The first execution model I would call the local execution model, this model involves forming tasks and flows and then executing them inside an application, that application is running for the duration of the workflow (although if it crashes it can re-establish the task and flows that it was doing and attempt to resume them). This could also be what openstack projects would call the 'conductor' approach where nova, ironic, trove have a conductor which manages these long-running actions (the conductor is alive/running throughout the duration of these workflows, although it may be restarted while running). The restarting + resuming part is something that openstack hasn't handled so gracefully currently, typically requiring either some type of cleanup at restart (or by operations), with taskflow using this model the resumption part makes it possible to resume from the last saved state (this connects into the persistence model that taskflow uses, the state transitions, how execution occurrs itself...).

The second execution model is an extension of the first, whereby there is still a type of 'conductor' that is managing the life-time of the workflow, but instead of locally executing tasks in the conductor itself tasks are now executed on remote-workers (see http://tinyurl.com/lf3yqe4
). The engine currently still is 'alive' for the life-time of the execution, although the work that it is doing is relatively minimal (since its not actually executing any task code, but proxying those requests to others works). The engine while running does the conducting of the remote-workers (saving persistence details, doing state-transtions, getting results, sending requests to workers…).

These two execution models are special cases of what you call “lazy execution model” (or passive as we call it). To illustrate this idea we can take a look at the first sequence diagram at [0], we basically will see the following interaction:

1) engine --(task)--> queue --(task)--> worker
2) execute task
3) worker --(result)--> queue --(result)--> engine

This is how TaskFlow worker based model works.

If we loosen the requirement in 3) and assume that not only worker can send a task result back to engine we’ll get our passive model. Instead of worker it can be anything else (some external system) that knows how to make this call. A particular way is not too important, it can be a direct message or it can be hidden behind an API method. In Mistral it’s now a REST API method however we’re about to decouple engine from REST API so that engine is a standalone process and listens to a queue. So worker-based model is basically the same with the only strict requirement that only worker sends a result back.

In order to implement local execution model on top of “lazy execution model” we just need to abstract a transport (queue) so that we can use an in-process transport. That’s it. It’s what Mistral already has implemented. Again, we see that “lazy execution model” is more universal.

IMO this “lazy execution model” should be the main execution model that TaskFlow supports, others can be easily implemented on top of it. But the opposite assertion is wrong. IMO this is the most important obstacle in all our discussions, the reason why we don’t always understand each other well enough. I know it may be a lot of work to shift a paradigm in TaskFlow team but if we did that we would get enough freedom for using TaskFlow in lots of cases.

Everything is a lot of work ;) That’s just how it goes. I think we work through it then it's all fine in the end.

I think some of this is being resolved/discussed @ http://tinyurl.com/k3s2gmy


Let me know what you think. I might have missed something.

=== HA ===

So this is an interesting question, and to me is strongly connected to how your engines are executing (and the persistence and state-transitions that they go through while running). Without persistence of state and transitions there is no good way (a bad way of course can be created, by just redoing all the work, but that's not always feasible or the best option) to accomplish resuming in a sane manner and there is also imho no way to accomplish any type of automated HA of workflows.

Sure, no questions here.

Let me describe:

When you save the states of a workflow and any intermediate results of a workflow to some database (for example) and the engine (see above models) which is being used (for example the conductor type from above) the application containing that engine may be prone to crashes (or just being powered off due to software upgrades...). Since taskflows key primitives were made to allow for resuming when a crash occurs, it is relatively simple to allow another application (also running a conductor) to resume whatever that prior application was doing when it crashed. Now most users of taskflow don't want to have to do this resumption manually (although they can if they want) so it would be expected that the other versions of that application would be running would automatically 'know' how to 'take-over' the work of the failed application. This is where the concept of the taskflows 'jobboard' (http://tinyurl.com/klg358j) comes into play, where a jobboard can be backed by something like zookeeper (which provides notifications of lock lose/release to others automatically). The jobboard is the place where the other applications would be looking to 'take-over' the other failed applications work (by using zookeeper 'atomic' primitives designed for this type of usage) and they would also be releasing the work back for others to 'take-over' when there own zookeeper connection is lost (zookeeper handles this this natively).

Instead of inventing this jobboard built with Zookeeper which is not always welcome in the community, why not just use a queue with message acknowledgements? In case if engine instance is processing a task and it fails a message broker just resubmits the corresponding message automatically. This is a standard, e.g. RabbitMQ feature. Am I missing something important?

Zookeeper is just one implementation, there is talks of redis, raft and others (that’s the reason the jobboard concept is an abstraction, not a implementation). The problem with MQ and others is they don't offer atomic locking +liveness+ownership semantics (afaik). That’s the difference, if a engine (or worker even) acks a messaging saying it owns the work in the given message, and the engine/woeker crashes (power-off) how does it resubmit the work (timeouts?), how can that happen automatically? With zookeeper or other systems that maintain liveness and ownership as built-in primitives they can offer these semantics (zookeeper concept of watches, locks, other recipes<https://github.com/python-zk/kazoo/tree/master/kazoo/recipe>…). To me that’s the difference between a MQ and something like zookeeper (or similar/like system), zookeeper provides said semantics natively, message queues are message queues.



Renat Akhmerov
@ Mirantis Inc.

[0] https://etherpad.openstack.org/p/mistral-engine-overview-and-proposal


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140401/4fdf7a0e/attachment-0001.html>