Open Stack

Hi Ceilo Folks,

TL;DR: just wanted to get feedback on some initial ideas on how
we might accommodate different conduits of publication from the
ceilo pollsters and notification handlers.

Currently, we're obviously hard-coded to always publish via the
message bus, so the idea is to support alternative/additional
publication routes, according to the following outline:

 - a ceilo notification handler or pollster can be associated
   via config with a chain of zero or more Publishers

 - each Publisher knows now to emit their representation over some
   conduit (AMQP notification, CW PutMetricData call, statsd UDP
   packet, or whatever)

 - a ceilometer notification handler or pollster provides a
   Transformer corresponding to each of its Publishers, which
   knows how to transform domain-specific raw data into the
   relevant representation

 - the batching of transformer output is up to the Publisher

 - each Publisher has an independent frequency, so not all
   Publishers are called to publish every measurement

So for example the compute.libvirt.DiskIOPollster would provide a
transformer to map the raw disk stats reported by libvirt into
DiskWriteOps, DiskWriteBytes, DiskReadOps & DiskReadBytes metric
datapoints for CW, also indicating that this should be published
very minute. This CWPublisher would handle batching these data
as a single PutMetricData call.

The DiskIOPollster would also provide a second Transformer to
mediate between the raw stats format and the ceilo Counter
representation for metering.  Additionally, this second Publisher
could be configured with a longer period (as befits metering data
likely to be consumed by a billing engine).

Now the frequency aspect is a bit awkward. For notifications,
the frequency of the "exists" style of usage audit is out of our
hands, depending usually on the configuration of the emitter
(e.g. the nova instance_usage_audit_period config). Whereas ceilo
config direct controls only the frequency of polling. So one idea 
would be for a publisher's frequency only to impact on the periods
under ceilo control - i.e. we'd accept and publish notifications
whenever we get them, whereas the polling cycle would be determined
by the greatest common divisor of all the configured publisher
periods for each set of related pollsters (the config would probably
have to be restricted so to ensure a sane GCD). 

Also, I'm assuming that an alternative publication route would
generally be used when the target endpoint is not the ceilo collector
& metering store ... i.e. non-standard publisher would most often
be used in order to hive off the data to an alternative endpoint
or store, such as a CW metrics store, or say to be dumped as RRD 
files to be consumed by graphite.

Thoughts?

Cheers,
Eoghan