Open Stack

On Mon, Jul 07, 2014 at 02:13:57AM -0400, Eoghan Glynn wrote:
> 
> 
> > In current Alarm implementation, Ceilometer will send back Heat an
> > 'alarm' using the pre-signed URL (or other channel under development).
> 
> By the other channel, do you mean the trusts-based interaction?

Yes, Sir. Trusts and redelegated trusts.

> We discussed this at the mid-cycle in Paris last week, and it turns out
> there appear to be a few restrictions on trusts that limit the usability
> of this keystone feature, specifically:
> 
>  * no support for cross-domain delegation of privilege (important as
>    the frontend stack user and the ceilometer service user are often
>    in different domains) 
> 
>  * no support for creating a trust based on username+domain as opposed
>    to user UUID (the former may be predictable at the time of config
>    file generation, whereas the latter is less likely to be so)
> 
>  * no support for cascading delegation (i.e. no creation of trusts from
>    trusts)
> 
> If these shortcomings are confirmed by the domain experts on the keystone
> team, we're not likely to invest further time in trusts until some of these
> issues are addressed on the keystone side.

Yes, agreed.  Let's look forward to some work from Keystone team then.

> > The alarm carries a payload that looks like:
> > 
> >  {
> >    alarm_id: ID
> >    previous: ok
> >    current: alarm
> >    reason: transision to alarm due to n samples outside thredshold,
> >            most recent: ....
> >    reason_data: {
> >      type: threshold
> >      disposition: inside
> >      count: x
> >      most_recent: value
> >    }
> >  }
> > 
> > While this data structure is useful for some simple use cases, it can be
> > enhanced to carry more useful data.  Some usage scenarios are:
> > 
> >  - When a member of AutoScalingGroup is dead (e.g. accidently deleted),
> >    Ceilometer can detect this from a event with count='instance',
> >    event_type='compute.instance.delete.end'.  If an alarm created out of
> >    this event, the AutoScalingGroup may have a chance to recover the
> >    member when appropriate.  The requirement is for this Alarm to tell
> >    Heat which instance is dead.
> 
> Alarms in ceilometer may currently only be based on a statistics trend
> crossing a threshold, and not on the occurrence of an event such as
> compute.instance.delete.end.

Right.  I realized this after spending some more time understanding the
alarm-evaluator code.  Having 'Statistics' model to record (even the
last sample of) a field will be cumbersome.

> Near the end of the Icehouse cycle, there was an attempt to implement
> this style of notification-based alarming but the feature did not land.

After realizing 'Statistics' is not the ideal place for extension, I
took a step back and asked myself: "what am I really trying to get from
Ceilometer?" The answer seems to be an Alarm or Event, with some
informational fields telling me some context of such an Alarm or Event.
So I am now thinking of a EventAlarm in addition to ThresholdAlarm and
CombinationAlarm.  The existing alarms are all based on meter samples.
Such an event based alarm would be very helpful to implement features
like keeping members of a AutoScalingGroup (or other Resource Group)
alive.

> Another option would be for Heat itself to consume notifications and/or
> periodically check the integrity of the autoscaling group via nova-api,
> to ensure no members have been inadvertently deleted.

Yes. That has been considered by the Heat team as well.  The only
concern regarding directly subscribing to notification and then do
filtering sounds a duplicated work already done in Ceilometer. From the
use case of convergence, you can guess that this is acutally not limited
to the auto-scaling scenario.

> This actually smells a little some of the requirements driving the
> notion of "convergence" in Heat:
> 
>   https://review.openstack.org/#/c/95907/6/specs/convergence.rst
> 
> TL;DR: make reality the source the truth in Heat, as opposed to the
>        approximation of reality expressed in the template
> 
> >  - When a VM connected to multiple subnets is experiencing bandwidth
> >    problem, an alarm can be generated telling Heat which subnet is to be
> >    checked.
> 
> Would such a bandwidth issue be suitable for auto-remediation by the
> *auto*scaling logic?
> 
> Or would it require manual intervention?

As I have noted above, getting notified by physical resource state
changes and then reacting properly is THE requirement.  It is beyond
what auto-scaling does today.  There are cases where manual intervention
is needed, while there are other cases where Heat can handle given
sufficient information.

> > We believe there will be many other use cases expecting an alarm to
> > carry some 'useful' information beyond just a state transition. Below is
> > a proposal to solve this.  Any comments are welcomed.
> > 
> > 1. extend the alarm with an optional parameter, say, 'output', which is
> >    a map or an equivalent representation.  A user can specify some
> >    key=value pairs using this parameter, where 'key' is a convenience
> >    for user and value is used to specify a field from a Sample whose
> >    value will be filled  in here.
> > 
> >    e.g. --output instance=metadata.instance_id;timestamp=timestamp
> 
> While such additional context may be useful, I'm not sure your examples
> would apply in general because:
> 
>  * there wouldn't be a *single* distinguished instance ID that caused
>    the alarm statistic to go over-threshold (as the cpu_util or whatever
>    metric is aggregated across the entire autoscaling group in the alarm
>    evaluation)
> 
>  * there wouldn't be a discrete timestamp when the statistic crossed the
>    alarm threshold due to perioidization and sampling effects

Okay.  I admit that if the alarm is evaluated based on Statistics, these
are all true concerns.  I didn't quite realize that before.  What do you
think if Ceilometer provides an EventAlarm then?  If Alarm is generated
from an Event, then the above context can be extracted, at least by
tweaking event_definitions.yaml?

> > 2. extend the Ceilometer alarm-evaluator service, so that when an alarm
> >    is seen requiring output values, it will try matching the 'value'
> >    specified above to the fields in a sample, and replace the output
> >    entry with 'key=<real_value>'.
> > 
> >    e.g. "output": {
> >           "instance": "bd56bb53-d07f-49a6-8f60-6f8ef1336060",
> > 	  "timestamp": "2014-07-0102: 21: 13.002155",
> > 	}
> > 
> >    The above data is passed back to the alarm_url as part of its
> >    existing payload.
> > 
> >    If alarm-evaluator cannot find a matching field, it can fill in an
> >    empty string, or just "None".
> > 
> > 3. extend the OS::Ceilometer::Alarm resource type in Heat so that an
> >    optional property (say, 'output') of type map can be used to specify
> >    what are expected from the Alarm.
> 
> And would the logic to consume such additional context be baked into
> the heat.engine.resources.autoscaling module?
> 
> Or would that be plugable somehow?

We have an interest to improve the AutoScalingGroup resource so that
member failures can be detected and handled properly.  This may warrant
a specs in Heat project.

> > Since it is an additional field in the 'details' argument, the impact to
> > existing Heat template/users will be negligible.  However, the
> > expressive power of carrying back additional fields would be a great
> > help to some scenarios we yet to know.
> > 
> > Because this is a cross-project proposal, comments from both communities
> > are valuable and thus appreciated.  If it is a viable approach, should
> > we raise two specs in both projects repectively?
> 
> I'm unconvinced as yet as to the viability, on the basis of my comments
> above. Though I'll keep an open mind with regard to your responses.
> 
> Thanks,
> Eoghan
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>