Open Stack

Just some random thoughts below ...

On Tue, Jul 01, 2014 at 03:47:03PM -0400, Mike Spreitzer wrote:
> In AWS, an autoscaling group includes health maintenance functionality --- 
> both an ability to detect basic forms of failures and an ability to react 
> properly to failures detected by itself or by a load balancer.  What is 
> the thinking about how to get this functionality in OpenStack?  Since 

We are prototyping a solution to this problem at IBM Research - China
lab.  The idea is to leverage oslo.messaging and ceilometer events for
instance (possibly other resource such as port, securitygroup ...)
failure detection and handling.

> OpenStack's OS::Heat::AutoScalingGroup has a more general member type, 
> what is the thinking about what failure detection means (and how it would 
> be accomplished, communicated)?

When most OpenStack services are making use of oslo.notify, in theory, a
service should be able to send/receive events related to resource
status.  In our current prototype, at least host failure (detected in
Nova and reported with a patch), VM failure (detected by nova), and some
lifecycle events of other resources can be detected and then collected
by Ceilometer.  There is certainly a possibility to listen to the
message queue directly from Heat, but we only implemented the Ceilometer
centric approach.

> 
> I have not found design discussion of this; have I missed something?
> 
> I suppose the natural answer for OpenStack would be centered around 
> webhooks.  An OpenStack scaling group (OS SG = OS::Heat::AutoScalingGroup 
> or AWS::AutoScaling::AutoScalingGroup or OS::Heat::ResourceGroup or 
> OS::Heat::InstanceGroup) could generate a webhook per member, with the 
> meaning of the webhook being that the member has been detected as dead and 
> should be deleted and removed from the group --- and a replacement member 
> created if needed to respect the group's minimum size.  

Well, I would suggest we generalize this into a event messaging or
signaling solution, instead of just 'webhooks'.  The reason is that
webhooks as it is implemented today is not carrying a payload of useful
information -- I'm referring to the alarms in Ceilometer.

There are other cases as well.  A member failure could be caused by a 
temporary communication problem, which means it may show up quickly when
a replacement member is already being created.  It may mean that we have
to respond to an 'online' event in addition to an 'offline' event?

> When the member is 
> a Compute instance and Ceilometer exists, the OS SG could define a 
> Ceilometer alarm for each member (by including these alarms in the 
> template generated for the nested stack that is the SG), programmed to hit 
> the member's deletion webhook when death is detected (I imagine there are 
> a few ways to write a Ceilometer condition that detects instance death). 

Yes.  Compute instance failure can be detected with a Ceilometer plugin.
In our prototype, we developed a Dispatcher plugin that can handle
events like 'compute.instance.delete.end', 'compute.instance.create.end'
after they have been processed based on a event_definitions.yaml file.
There could be other ways, I think.

The problem here today is about the recovery of SG member.  If it is a
compute instance, we can 'reboot', 'rebuild', 'evacuate', 'migrate' it,
just to name a few options.  The most brutal way to do this is like what
HARestarter is doing today -- delete followed by a create.

> When the member is a nested stack and Ceilometer exists, it could be the 
> member stack's responsibility to include a Ceilometer alarm that detects 
> the member stack's death and hit the member stack's deletion webhook. 

This is difficult.  A '(nested) stack' is a Heat specific abstraction --
recall that we have to annotate a nova server resource in its metadata
to which stack this server belongs.  Besides the 'visible' resources
specified in a template, Heat may create internal data structures and/or
resources (e.g. users) for a stack.  I am not quite sure a stack's death
can be easily detected from outside Heat.  It would be at least
cumbersome to have Heat notify Ceilometer that a stack is dead, and then
have Ceilometer send back a signal.

> There is a small matter of how the author of the template used to create 
> the member stack writes some template snippet that creates a Ceilometer 
> alarm that is specific to a member stack that does not exist yet.  

How about just one signal responder per ScalingGroup?  A SG is supposed
to be in a better position to make the judgement: do I have to recreate
a failed member? am I recreating it right now or wait a few seconds?
maybe I should recreate the member on some specific AZs?

If there is only one signal responder per SG, then the 'webhook' (or
resource signal, my preference) need to carry a payload indicating when
and which member is down. 

> I suppose we could stipulate that if the member template includes a 
> parameter with name "member_name" and type "string" then the OS OG takes 
> care of supplying the correct value of that parameter; as illustrated in 
> the asg_of_stacks.yaml of https://review.openstack.org/#/c/97366/ , a 
> member template can use a template parameter to tag Ceilometer data for 
> querying.  The URL of the member stack's deletion webhook could be passed 
> to the member template via the same sort of convention.  

I am not in favor of the per-member webhook design.  But I vote for an
additional *implicit* parameter to a nested stack of any groups.  It
could be an index or a name.

> When Ceilometer 
> does not exist, it is less obvious to me what could usefully be done.  Are 
> there any useful SG member types besides Compute instances and nested 
> stacks?  Note that a nested stack could also pass its member deletion 
> webhook to a load balancer (that is willing to accept such a thing, of 
> course), so we get a lot of unity of mechanism between the case of 
> detection by infrastructure vs. application level detection.
> 

I'm a little bit concerned about passing the member deletion webhook to
LB.  Maybe we need to rethink about this: do we really want to bring
application level design considerations down to the infrastructure level?

Some of the detection work might be covered by the observer engine specs
that is under review.  My doubt about it is about how to make it "listen
only to what need to know while ignore everything else".

> I am not entirely happy with the idea of a webhook per member.  If I 
> understand correctly, generating webhooks is a somewhat expensive and 
> problematic process.  What would be the alternative?

My understanding is that the webhooks' problem is not about cost, it is
more about authentication and flexibility.  Steve Hardy and Thomas Herve
are already looking into the authentication problem.

> Mike

> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev