Open Stack

Better late than never...

On 16/12/16 08:57, Steven Hardy wrote:
>
> I know we've previously tried to steer execute/run type actions to signal
> driven interfaces (and I myself have opposed these kinds of resources in
> the past, to be honest).  However, I can't currently see a better way to
> handle this requirement, and given it may be pretty easy to fix (refactor
> handle_signal and add a boolean to each handle_foo function), I think this
> discussion warrants revisiting.

I think we need to get away from thinking of this as "I just need to run 
a script in the middle of Heat's workflow, let me do it!". That's verb 
thinking and that's the reason I -1'd 
https://review.openstack.org/267770 originally.

We need to think of it as "I have some external thing that I need to 
manage, but I can't teach Heat about it because it means nothing to my 
cloud operator". That's noun thinking and something we can actually work 
with.

For that reason I started talking to a few folks a while back about 
implementing an equivalent of CloudFormation's "CustomResource"[1]. 
Basically, it sends a message to an SNS topic (c.f. Zaqar notification) 
and waits for a success/failure response. We have the technology now to 
trigger Mistral workflows from Zaqar messages.[2] It's clear that this 
is both conceptually sound and solves a number of problems with the best 
available solution, which is triggering workflows via Zaqar 
notifications from user hooks:
  - There's no way to fail a particular resource.
  - The Zaqar queue has to be passed in to the environment when creating 
the stack. This means that the queue has to be created outside of the stack.
  - All stacks in the tree share the same queue for hook notifications.

One reason that CloudFormation implements CustomResource as a message to 
a topic is that it allows integration with third-party service 
providers. That's a valid use case, but I think there's an equally or 
more valid use case for integrating with stuff in your own tenant that 
is just under the end user's control rather than the operators. For that 
case, it makes sense to cut out the Zaqar middleman and just trigger the 
Mistral execution directly.

So I am on board with a Mistral execution resource, BUT:

  * I don't think we should call it a Mistral Execution. I think we 
should give it a name that indicates it's managing some external thing 
by executing Mistral workflows.
  * We need to learn the lesson of SoftwareComponent and include _all_ 
the different phases of the lifecycle in the _same resource_. The user 
should not have to assemble the different phases (Create/Update/Delete) 
of their external resource using discrete workflows, like they do with 
SoftwareConfigs. It's too much to ask them to understand how to build 
the dependency graphs correctly - it took me over a year to work it 
out[3] and that was my whole job.
  * We should also learn the lesson of 
https://launchpad.net/bugs/1595040 and implement a mechanism to allow 
the template author to select (in advance) between update-in-place and 
update-replace for any given update from the get-go.

I also left similar comments on https://review.openstack.org/#/c/420664/ 
but I wanted to go into the background in more depth here.

We may _also_ want to implement something like CustomResource in the 
future, but I agree that this is the higher priority.

cheers,
Zane.

[1] 
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html
[2] 
http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Zaqar::MistralTrigger
[3] 
https://review.openstack.org/gitweb?p=openstack/heat.git;a=commitdiff;h=8b732241df6007c3005b14413ee1fe047eb4d108