Open Stack

Joshua,

On Apr 17, 2013, at 10:37 PM, Joshua Harlow <harlowja at yahoo-inc.com> wrote:

> I'd be very interested in having 'Scheduler & Workflow Service' part there
> be a library.
> 
> Pretty much every application is a workflow in some way, and using said
> library in nova for the orc work there would be very neat.

We had an unconference session today when we discussed this concept:

https://wiki.openstack.org/wiki/Convection

The idea is that this could start as a library in Heat, and then be moved to Oslo upon achievement of a reasonable level of maturity.

> As long as it doesn't change the use-case that heat desires of course (or
> overload that use-case and make everything complex when it doesn't need to
> be)…

It should actually simplify Heat.

> It would be very neat to use those 2 services as a library, where I can
> submit arbitrary code to (the state transitions that nova does) and have
> it handle calling those states, coordinating, rolling back (or at least
> calling a method to rollback, since rollback is usually very specific to
> what has occurred). Basically submitting jobs, but not via a DSL/CEP (or
> the like), not via a model interpreter, but directly via code. Is there
> any documentation on how heat handles task resumption (if 1 engine fails,
> another should be able to continue the work), how are said engines made HA
> and reliable…

At the very least, we should expect a solution that will allow you to submit a callback in some form (possibly a web hook, or a function) so that you can act upon state transitions in the workflow. There are ways to deal with rollback. Chances are that a rather simple early implementation would shift that to the client, but there are way to deal with it in the service too.

> Such a engine library would make sense as the core 'engine' for a lot of
> the openstack core projects imho.

Yes, if the vision is achieved, you would use Heat for Orchestration purposes, and Convection for simple task executions. The first use case for Convection would be a task that has a start, a sequence of tasks, followed by a termination. This could be very useful in Nova when you care about kicking off a workflow and be able to get a callback upon completion of the job/task.

For jobs that would benefit from parallel execution and coordination of multiple tasks, you would use Heat as an Orchestration instrument, which would in turn use multiple workflows.

> Thoughts?

The first implementation is likely to be library within the Heat project that the Heat engine uses. Next, an Oslo library, and then a standalone service with a REST API that uses the Oslo library for general purpose uses from applications outside Openstack, perhaps those deployed by Heat, etc. This gives us the opportunity to offer the service in an fault tolerant HA configuration for use cases that require reliable workflow execution. If you are interested in contributing, I urge you to connect with Keith Bray <keith.bray at racksopace.com> to coordinate efforts.

Cheers,

Adrian

> 
> On 4/17/13 2:47 PM, "Zane Bitter" <zbitter at redhat.com> wrote:
> 
>> Thanks Adrian!
>> 
>> We should probably make it clear for those who can't be at Summit that
>> this is not intended to be a record of the design summit discussions.
>> I'd describe this as a snapshot of Adrian's evolving vision for Heat,
>> taken after the design summit sessions. In particular, I think it
>> reflects a couple of misconceptions about the current architecture of
>> Heat, and it also contains some stuff that hasn't been discussed at all
>> at Summit (e.g. the CEP part... btw my initial reaction to this is that
>> it should probably talk to Ceilometer rather than the Autoscaling
>> service).
>> 
>> That said, there certainly *is* consensus that we want to evolve Heat
>> toward being able to orchestrate whole applications rather than just
>> services. We'll be doing further work throughout the week and
>> subsequently on the mailing list to refine some of these architectural
>> ideas, and we're looking forward to working with Adrian and his team at
>> Rackspace to make it happen :)
>> 
>> cheers,
>> Zane.
>> 
>> On 16/04/13 01:56, Adrian Otto wrote:
>>> Heaters,
>>> 
>>> I attended the various sessions at the Design Summit today in Portland,
>>> and assembled as many of the ideas for future planning as I could.  For
>>> the benefit of those who are not attending, or who were not in these
>>> sessions, I created this Wiki page to express what I think is an early
>>> consensus on where we could take things. Let's tweak this if it's not a
>>> good direction.
>>> 
>>> https://wiki.openstack.org/wiki/Heat/Vision
>>> 
>>> Keith will be doing an Unconference session on the Workflow Service
>>> ideaŠ I believe on Wednesday afternoon.
>>> 
>>> Thanks,
>>> 
>>> Adrian
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>> 
>> 
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev