Open Stack

On 04/07/2015 10:10 AM, gordon chung wrote:
> 
>> - Specific user oriented log messages (distinct from our normal
> operator logs)
>> - Deprecation messages (if they are using old resource
> properties/template features)
>> - Progress and resource state changes (an application doesn't want to
> poll an api for a state change)
>> - Automated actions (autoscaling events, time based actions)
>> - Potentially integrated server logs (from in guest agents)
> 

Angus mentions that we want "user messages" but I'd argue that an events
interface (not to be confused with our current stack events) would be a
great fit. We could define schemas (building on API-WG's error message
guide[1]) for heat events so users can programmatically interact with
heat w/o polling all the time.

I think moving slightly up the abstraction ladder from "here's a log
message" to "here's a structured event with extra metadata too" would be
great, and do a better job helping users than unstructured log messages.
The difference between the AWS options is instructive here because the
older service (CloudWatch) is log-line oriented*, while the newer
service (CloudTrail) provides structured events.

Tooling in general seems to be moving towards richer event data as well.
The logging tools (Loggly/Logstash/PaperTrail/zillions of others) are
intended to take your unstructured logs and turn them into events, so
why not have Heat output structured events that we can present to the
user with Ceilometer rather than sending log lines (through syslog or
otherwise) and using tooling to reassemble them into events later.

TL;DR: I think what we really want is a place to send and react to
*events*. Logs are a way to do that, of course, but the Ceilometer way
sounds pretty attractive.

* CloudWatch has you send unstructured log messages, then build filters
to parse them into quantifiable events, then set alarms on those metrics.

> is the idea that Heat would build events from the logs or would you want
> to send the log messages to another service to be process? so for
> example, Nova doesn't send all logs messages to the queue but they do
> send a set of messages relating to certain actions and errors that occur
> (beyond just CRUD events). as the use cases above seem to target
> specific actions/logs and not all logs, i would think the processing
> could be done on the initiators service end and not on the consumer end.
> 
> to give an example of what Ceilometer is capable of; Ceilometer
> currently takes JSON messages from the MQ from *most* services and from
> there we capture the entire raw notification and index on a select set
> of key-value pairs. i think it's entirely possible to take in non-json
> log messages and build an indexer around that if needed.
> 

I don't think it would be too hard for us to package up events like
stack state transitions, failures (with as much debug info as is
reasonable), autoscaling actions, etc.

(please pardon any gross terminology-mangling, I'm not very familiar
with Ceilometer)

We already use notifications to send what I'd term "sparse events" that
only include the affected stack ID, meaning there's not much to slice on
in Ceilometer.

This could be extended to richer JSON events that include the stack,
resources affected in the update, stats like "num-deleted-resources" or
"num-replaced-resources", autoscaling actions, and info about stack errors.

Is there a way for users as-is to view those raw notifications, not just
the indexed k/v pairs?

Thanks,
Ryan

[1]: https://review.openstack.org/#/c/167793/

-- 
Ryan Brown / Software Engineer, Openstack / Red Hat, Inc.