Open Stack

It looks like this discussion involves many of the issues faced when
developing the Curvature & Donabe frameworks, which were presented at the
Portland Summit - slides and video here:

http://www.openstack.org/summit/portland-2013/session-videos/presentation/i
nteractive-visual-orchestration-with-curvature-and-donabe

Much of the work on the Donabe side revolved around defining a simple
JSON-based API for describing the sorts of virtual application templates
being discussed. All of the code for both Curvature and Donabe has
recently been made open source and is available here:

http://ciscosystems.github.io/curvature/

http://ciscosystems.github.io/donabe/

It looks like some of the ground covered by these projects can be helpful
to this discussion.

John Davidge
jodavidg at cisco.com



>---------- Forwarded message ----------
>From: Thomas Spatzier <thomas.spatzier at de.ibm.com>
>Date: Wed, Oct 9, 2013 at 12:40 AM
>Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
>proposal for workflows
>To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
>
>
>Excerpts from Clint Byrum's message
>
>> From: Clint Byrum <clint at fewbar.com>
>> To: openstack-dev <openstack-dev at lists.openstack.org>,
>> Date: 09.10.2013 03:54
>> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
>> proposal for workflows
>>
>> Excerpts from Stan Lagun's message of 2013-10-08 13:53:45 -0700:
>> > Hello,
>> >
>> >
>> > That is why it is necessary to have some central coordination service
>which
>> > would handle deployment workflow and perform specific actions (create
>VMs
>> > and other OpenStack resources, do something on that VM) on each stage
>> > according to that workflow. We think that Heat is the best place for
>such
>> > service.
>> >
>>
>> I'm not so sure. Heat is part of the Orchestration program, not
>>workflow.
>>
>
>I agree. HOT so far was thought to be a format for describing templates in
>a structural, declaritive way. Adding workflows would stretch it quite a
>bit. Maybe we should see what aspects make sense to be added to HOT, and
>then how to do workflow like orchestration in a layer on top.
>
>> > Our idea is to extend HOT DSL by adding  workflow definition
>capabilities
>> > as an explicit list of resources, components¹ states and actions.
>States
>> > may depend on each other so that you can reach state X only after
>you¹ve
>> > reached states Y and Z that the X depends on. The goal is from initial
>> > state to reach some final state ³Deployed².
>> >
>
>We also would like to add some mechanisms to HOT for declaratively doing
>software component orchestration in Heat, e.g. saying that one component
>depends on another one, or needs input from another one once it has been
>deployed etc. (I BTW started to write a wiki page, which is admittedly far
>from complete, but I would be happy to work on it with interested folks -
>https://wiki.openstack.org/wiki/Heat/Software-Configuration-Provider).
>However, we must be careful not to make such features too complicated so
>nobody will be able to use it any more. That said, I believe we could make
>HOT cover some levels of complexity, but not all. And then maybe workflow
>based orchestration on top is needed.
>
>>
>> Orchestration is not workflow, and HOT is an orchestration templating
>> language, not a workflow language. Extending it would just complect two
>> very different (though certainly related) tasks.
>>
>> I think the appropriate thing to do is actually to join up with the
>> TaskFlow project and consider building it into a workflow service or
>tools
>> (it is just a library right now).
>>
>> > There is such state graph for each of our deployment entities
>>(service,
>> > VMs, other things). There is also an action that must be performed on
>each
>> > state.
>>
>> Heat does its own translation of the orchestration template into a
>> workflow right now, but we have already discussed using TaskFlow to
>> break up the orchestration graph into distributable jobs. As we get more
>> sophisticated on updates (rolling/canary for instance) we'll need to
>> be able to reason about the process without having to glue all the
>> pieces together.
>>
>> > We propose to extend HOT DSL with workflow definition capabilities
>where
>> > you can describe step by step instruction to install service and
>properly
>> > handle errors on each step.
>> >
>> > We already have an experience in implementation of the DSL, workflow
>> > description and processing mechanism for complex deployments and
>believe
>> > we¹ll all benefit by re-using this experience and existing code,
>>having
>> > properly discussed and agreed on abstraction layers and distribution
>>of
>> > responsibilities between OS components. There is an idea of
>implementing
>> > part of workflow processing mechanism as a part of Convection
>>proposal,
>> > which would allow other OS projects to benefit by using this.
>> >
>> > We would like to discuss if such design could become a part of future
>Heat
>> > version as well as other possible contributions from Murano team.
>> >
>>
>> Thanks really for thinking this through. Windows servers are not unique
>in
>> this regard. Puppet and chef are pretty decent at expressing single-node
>> workflows but they are awkward for deferring control and resuming on
>other
>> nodes, so I do think there is a need for a general purpose distributed
>> workflow definition tool.
>>
>> I'm not, however, convinced that extending HOT would yield a better
>> experience for users. I'd prefer to see HOT have a well defined
>>interface
>> for where to defer to external workflows. Wait conditions are actually
>> decent at that, and I'm sure with a little more thought we can make them
>> more comfortable to use.
>
>Good discussion to have, i.e. what extensions we would need in HOT for
>making HOT alone more capable, and what we would need to hook it up with
>other orchestration like workflows.
>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>