Open Stack

Hi Clint,

thanks for writing this down. This is a really interesting use case and
feature, also in relation to what was recently discussed on rolling
updates.

I have a couple of thoughts and questions:

1) The overall idea seems clear to me but I have problems understanding the
detailed flow and relation to template definitions and metadata. E.g. in
addition to the examples you gave in the linked etherpad, where would the
script or whatever sit that handles the update etc.

2) I am not a big fan of CFN WaitConditions since they let too much
programming shine thru in a template. So I wonder whether this could be
made more transparent to the template writer. The underlying mechanism
could still be the same, but maybe we could make the template look cleaner.
For example, what Steve Baker is doing for software orchestration also uses
the underlying mechanisms but does not expose WaitConditions in templates.

3) Has the issue of how to express update policies on the rolling updates
thread been resolved? I followed that thread but seems like there has not
been a final decision. The reason I am bringing this up is because I think
this is related. You are suggesting to establish a new top-level section
'action_hooks' in a resource. Rendering this top-level in the resource is a
good thing IMO. However, since this is related to updates in a way (you
want to react to any kind of update event to the resource's state), I
wonder if those hooks could be attributes of an update policy. UpdatePolicy
in CFN is also a top-level section in a resource and they seem to provide a
default one like the following (I am writing this in snake case as we would
render it in HOT:

resources:
  autoscaling_group1:
    type: AWS::AutoScaling::AutoScalingGroup
    properties:
      # the properties ...
    update_policy:
      auto_scaling_rolling_update:
        min_instances_in_server: 1
        max_batch_size: 1
        pause_time: PT12M5S

(I took this from the CFN user guide).
I.e. an update policy already is a complex data structure, and we could
define additional types that include the resource hooks definitions you
need. ... I don't fully understand the connection between 'actions' and
'path' in your etherpad example yet, so cannot define a concrete example,
but I hope you get what I wanted to express.

4) What kind of additional metadata for the update events are you thinking
about? For example, in case this is done in an update case with a batch
size of > 1 (i.e. you update multiple members in a cluster at a time) -
unless I put too much interpretation in here concerning the relation to
rolling updates - you would probably want to tell the server a black list
of servers to which it should not migrate workload, because they will be
taken down as well.


As I said, just a couple of thoughts, and maybe for some I am just
mis-understanding some details.
Anyway, I would be interested in your view.

Regards,
Thomas


Clint Byrum <clint at fewbar.com> wrote on 11/02/2014 06:22:54:

> From: Clint Byrum <clint at fewbar.com>
> To: openstack-dev <openstack-dev at lists.openstack.org>
> Date: 11/02/2014 06:30
> Subject: [openstack-dev] [Heat] in-instance update hooks
>
> Hi, so in the previous thread about rolling updates it became clear that
> having in-instance control over updates is a more fundamental idea than
> I had previously believed. During an update, Heat does things to servers
> that may interrupt the server's purpose, and that may cause it to fail
> subsequent things in the graph.
>
> Specifically, in TripleO we have compute nodes that we are managing.
> Before rebooting a machine, we want to have a chance to live-migrate
> workloads if possible, or evacuate in the simpler case, before the node
> is rebooted. Also in the case of a Galera DB where we may even be running
> degraded, we want to ensure that we have quorum before proceeding.
>
> I've filed a blueprint for this functionality:
>
> https://blueprints.launchpad.net/heat/+spec/update-hooks
>
> I've cobbled together a spec here, and I would very much welcome
> edits/comments/etc:
>
> https://etherpad.openstack.org/p/heat-update-hooks
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>