Open Stack

On 18/04/13 07:37, Joshua Harlow wrote:
> I'd be very interested in having 'Scheduler & Workflow Service' part there
> be a library.
>
> Pretty much every application is a workflow in some way, and using said
> library in nova for the orc work there would be very neat.

+1
>
> As long as it doesn't change the use-case that heat desires of course (or
> overload that use-case and make everything complex when it doesn't need to
> be)...

That shouldn't be a big issue. Heat's use case and Nova's are similar, 
except that we also need to run tasks in parallel. However, I'm in the 
process of delivering that feature in Heat now and the 'library' part of 
that code is very localised, so I don't think it would add much 
complexity for anyone not using it.
>
> It would be very neat to use those 2 services as a library, where I can
> submit arbitrary code to (the state transitions that nova does) and have
> it handle calling those states, coordinating, rolling back (or at least
> calling a method to rollback, since rollback is usually very specific to
> what has occurred). Basically submitting jobs, but not via a DSL/CEP (or
> the like), not via a model interpreter, but directly via code.

Yes, I very much like the idea of just having a library that takes 
Python code and runs it as a workflow, rather than having some sort of 
DSL from which you have to try to generate/locate the code each time. 
(That stuff would be the job of the proposed Convection service, which 
ideally could also be built on top of this library.)

> Is there
> any documentation on how heat handles task resumption (if 1 engine fails,
> another should be able to continue the work), how are said engines made HA
> and reliable...

This is an issue we haven't yet tackled in Heat - there was a Design 
Summit session about it, and it is on the agenda for Havana.

The easiest way forward, for both Heat and Nova, might be to mandate 
that the tasks are written to be idempotent and rely on their existing 
use of state stored in the database. Then we would need only one level 
of locking (to determine whether the workflow is still being run, or has 
died and we need to resume it or roll it back).

The more sophisticated approach is obviously to have it store the 
current progress in the database and have some sort of event triggered 
when a task completes that lets any other workflow engine process pick 
it up and start the next step. I'm not convinced that's feasible for a 
simple library though; a service with a DSL could do it, but I'd rather 
not go down that path.

cheers,
Zane.

>
> Such a engine library would make sense as the core 'engine' for a lot of
> the openstack core projects imho.
>
> Thoughts?