Open Stack

Excerpts from Stan Lagun's message of 2013-10-08 13:53:45 -0700:
> Hello,
> 
> I’m one of the engineer working on Murano project. Recently we started a
> discussion about Murano and Heat Software orchestration and I want to
> continue this discussion with more technical details.
> 
> In our project we do deployment of complex multi-instance Windows services.
> Those services usually require specific multi-VM orchestration that is
> currently impossible or at least not that easy to achieve with Heat. As you
> are currently doing HOT software orchestration design we would like to
> participate in HOT Software orchestration design and contribute into it, so
> that the Heat could address use-cases which we believe are very common.
> 
> For example here is how deployment of a SQL Server cluster goes:
> 
>    1.
> 
>    Allocate Windows VMs for SQL Server cluster
>    2.

Heat does this quite well. :)

> 
>    Enable secondary IP address from user input on all SQL Windows instances
>    3.

I don't understand what that means.

> 
>    Install SQL Server prerequisites on each node
>    4.
> 

Automating software installation is a very common aspect of most
configuration tools.

>    Choose a master node and install Failover Cluster on it
>    5.
> 

Leader election should be built in to services that take advantage of
it. External leader election is really hard without all of the
primitives built into the service. Otherwise you have to resort to
things like STONITH to avoid split brain. If Heat tries to do this, it
will just result in a complicated design with very few happy users.

>    Configure all nodes so that they know which one of them is the master
>    6.
> 

Anyway, Heat can do this quite well. The template author can just choose
one, and flag it as the master in the resource's Metadata. Point all
others at that one using Ref's to it.

"Configuring" nodes should be something an automated configuration tool
does well. It is pretty straight forward to have a tool fetch the
Metadata of a resource via the Heat API and feed it to a configuration
tool. Or you can even teach your config tool to read the Metadata.

>    Install SQL Server on all nodes
>    7.
> 

Same as before - any automated software installation tool should
suffice.

>    Initialize AlwaysOn on all nodes except for the master
>    8.
> 

I don't know what that means.

>    Initialize Primary replica
>    9.
> 
>    Initialize secondary replicas
> 

I also don't know what that entails, but I would suspect that it is the
kind of thing you do via the SQL server API once everything is done.

For that, you just need to poll a waitcondition and when all of the
nodes have signaled the waitcondition, run this.

> 
> All of the steps must take place in appropriate order depending on the
> state of other nodes. Some steps require an output from previous steps and
> all of them require some input parameters. SQL Server requires an Active
> Directory service in order to use Failover mechanism and installation of
> Active Directory with primary and secondary controllers is a complex
> workflow of its own.
> 
> That is why it is necessary to have some central coordination service which
> would handle deployment workflow and perform specific actions (create VMs
> and other OpenStack resources, do something on that VM) on each stage
> according to that workflow. We think that Heat is the best place for such
> service.
> 

I'm not so sure. Heat is part of the Orchestration program, not workflow.

> Our idea is to extend HOT DSL by adding  workflow definition capabilities
> as an explicit list of resources, components’ states and actions. States
> may depend on each other so that you can reach state X only after you’ve
> reached states Y and Z that the X depends on. The goal is from initial
> state to reach some final state “Deployed”.
> 

Orchestration is not workflow, and HOT is an orchestration templating
language, not a workflow language. Extending it would just complect two
very different (though certainly related) tasks.

I think the appropriate thing to do is actually to join up with the
TaskFlow project and consider building it into a workflow service or tools
(it is just a library right now).

> There is such state graph for each of our deployment entities (service,
> VMs, other things). There is also an action that must be performed on each
> state.

Heat does its own translation of the orchestration template into a
workflow right now, but we have already discussed using TaskFlow to
break up the orchestration graph into distributable jobs. As we get more
sophisticated on updates (rolling/canary for instance) we'll need to
be able to reason about the process without having to glue all the
pieces together.

> We propose to extend HOT DSL with workflow definition capabilities where
> you can describe step by step instruction to install service and properly
> handle errors on each step.
> 
> We already have an experience in implementation of the DSL, workflow
> description and processing mechanism for complex deployments and believe
> we’ll all benefit by re-using this experience and existing code, having
> properly discussed and agreed on abstraction layers and distribution of
> responsibilities between OS components. There is an idea of implementing
> part of workflow processing mechanism as a part of Convection proposal,
> which would allow other OS projects to benefit by using this.
> 
> We would like to discuss if such design could become a part of future Heat
> version as well as other possible contributions from Murano team.
> 

Thanks really for thinking this through. Windows servers are not unique in
this regard. Puppet and chef are pretty decent at expressing single-node
workflows but they are awkward for deferring control and resuming on other
nodes, so I do think there is a need for a general purpose distributed
workflow definition tool.

I'm not, however, convinced that extending HOT would yield a better
experience for users. I'd prefer to see HOT have a well defined interface
for where to defer to external workflows. Wait conditions are actually
decent at that, and I'm sure with a little more thought we can make them
more comfortable to use.