Open Stack

On 04/07/2015 02:34 PM, Sandy Walsh wrote:
> 
>> Tooling in general seems to be moving towards richer event data as well.
>> The logging tools (Loggly/Logstash/PaperTrail/zillions of others) are
>> intended to take your unstructured logs and turn them into events, so
>> why not have Heat output structured events that we can present to the
>> user with Ceilometer rather than sending log lines (through syslog or
>> otherwise) and using tooling to reassemble them into events later.
> 
> The trend in the monitoring space seems to be:
> 
> 1. Alarms are issued from Metrics as Events. 
>     (events can issue alarms too, but conventional alarming is metric based)
> 2. Multiple events are analyzed to produce Metrics (stream processing)
> 3. Go to Step 1
> 

Indeed. I sort of envisioned heat sending out events that are then
consumed both as metrics and by the user (where appropriate). In
StackTach I can see that being implemented as

                    /--> resource events ----> other tools
Heat --> Winchester ---> notifications stream    ------> user
                    \--> metrics stream --> alerts --/


>> TL;DR: I think what we really want is a place to send and react to
>> *events*. Logs are a way to do that, of course, but the Ceilometer way
>> sounds pretty attractive.
> 
> The difference is structured vs. unstructured data. Elasticsearch-based
> solutions tend to ignore structure by looking at keywords. Some solutions,
> like TopLog, infer a structure by gleaning regexs from logs. 
> 
> Events start as structured data. More so, we're looking at establishing 
> AVRO-based schema definitions on top of these events (slow progress).

Yeah, I'd really like to have a schema for Heat events so we can have a
single event stream and repackage events for different consumption goals
(metrics, notifications, programmatic interaction, etc).

> If anything we should consider changing the logging library to use structured 
> messages. Specifically:
> 
> log("The %(foo)s did %(thing)s" % {'foo':'x', 'thing':'action'})
> it would be
> log({'message':"The %(foo)s did %(thing)s", 'foo':'x', 'thing':'action'})
> 
> Which can still be formatted for conventional logs, but also sent as a
> notification or as a higher-level structure to things like ES, TopLog, etc.
> The driver can decide. 
> 
>> * CloudWatch has you send unstructured log messages, then build filters
>> to parse them into quantifiable events, then set alarms on those metrics.
> 
> Having to build filters is a relatively error-prone approach compared to the
> methods described above. 

I wasn't saying *we* should do the unstructured message + regex filters
strategy, I was just pointing out the CW solution for folks who hadn't
used it.

>>> [snip]
> 
> The Fujitsu team have already added logging support to Monasca (with an 
> elasticsearch backend) and HP is currently adding StackTach.v3 support for
> notification->event conversion as well as our Winchester event stream 
> processing engine. Also, this is based on Kafka vs. RabbitMQ, which has better
> scaling characteristics for this kind of data.

Oooh, I'll have a look into that, Kafka as an event bus sounds like a
good fit. I have the same concern Angus voiced earlier about Zaqar
though. What's the deployment of StackTach.v3 across OpenStack
installations? Is it mostly deployed for Helion/Rackspace, or are
smaller deployers using it as well?

> 
>> This could be extended to richer JSON events that include the stack,
>> resources affected in the update, stats like "num-deleted-resources" or
>> "num-replaced-resources", autoscaling actions, and info about stack errors.
> 
> Some of these sound more like a metrics than notifications. We should be 
> careful not to misuse the two. 

I think they're events, and have facets that are quantifiable as metrics
(num-replaced-resources on an update action) and that should be
user-visible (update is complete, or autoscaling actions taken).

>> Is there a way for users as-is to view those raw notifications, not just
>> the indexed k/v pairs?
> 
> In StackTach.v3 we ship the raw notifications to HDFS for archiving, but
> expose the reduced event via the API. The message-id links the two.
> 
> Lots more here: http://www.stacktach.com

Thanks! I'll have to read up.

-- 
Ryan Brown / Software Engineer, Openstack / Red Hat, Inc.