Open Stack

>Tooling in general seems to be moving towards richer event data as well.
>The logging tools (Loggly/Logstash/PaperTrail/zillions of others) are
>intended to take your unstructured logs and turn them into events, so
>why not have Heat output structured events that we can present to the
>user with Ceilometer rather than sending log lines (through syslog or
>otherwise) and using tooling to reassemble them into events later.

The trend in the monitoring space seems to be:

1. Alarms are issued from Metrics as Events. 
    (events can issue alarms too, but conventional alarming is metric based)
2. Multiple events are analyzed to produce Metrics (stream processing)
3. Go to Step 1

>TL;DR: I think what we really want is a place to send and react to
>*events*. Logs are a way to do that, of course, but the Ceilometer way
>sounds pretty attractive.

The difference is structured vs. unstructured data. Elasticsearch-based
solutions tend to ignore structure by looking at keywords. Some solutions,
like TopLog, infer a structure by gleaning regexs from logs. 

Events start as structured data. More so, we're looking at establishing 
AVRO-based schema definitions on top of these events (slow progress).

If anything we should consider changing the logging library to use structured 
messages. Specifically:

log("The %(foo)s did %(thing)s" % {'foo':'x', 'thing':'action'})
it would be
log({'message':"The %(foo)s did %(thing)s", 'foo':'x', 'thing':'action'})

Which can still be formatted for conventional logs, but also sent as a
notification or as a higher-level structure to things like ES, TopLog, etc.
The driver can decide. 

>* CloudWatch has you send unstructured log messages, then build filters
>to parse them into quantifiable events, then set alarms on those metrics.

Having to build filters is a relatively error-prone approach compared to the
methods described above. 

>> [snip]

The Fujitsu team have already added logging support to Monasca (with an 
elasticsearch backend) and HP is currently adding StackTach.v3 support for
notification->event conversion as well as our Winchester event stream 
processing engine. Also, this is based on Kafka vs. RabbitMQ, which has better
scaling characteristics for this kind of data.

>This could be extended to richer JSON events that include the stack,
>resources affected in the update, stats like "num-deleted-resources" or
>"num-replaced-resources", autoscaling actions, and info about stack errors.

Some of these sound more like a metrics than notifications. We should be 
careful not to misuse the two. 

>Is there a way for users as-is to view those raw notifications, not just
>the indexed k/v pairs?

In StackTach.v3 we ship the raw notifications to HDFS for archiving, but
expose the reduced event via the API. The message-id links the two.

Lots more here: http://www.stacktach.com