Open Stack

FYI relevant blueprints:

  https://blueprints.launchpad.net/ceilometer/+spec/multi-publisher
  https://blueprints.launchpad.net/ceilometer/+spec/publisher-counters-frequency

Cheers,
Eoghan

> Hi Ceilo Folks,
> 
> TL;DR: just wanted to get feedback on some initial ideas on how
> we might accommodate different conduits of publication from the
> ceilo pollsters and notification handlers.
> 
> Currently, we're obviously hard-coded to always publish via the
> message bus, so the idea is to support alternative/additional
> publication routes, according to the following outline:
> 
>  - a ceilo notification handler or pollster can be associated
>    via config with a chain of zero or more Publishers
> 
>  - each Publisher knows now to emit their representation over some
>    conduit (AMQP notification, CW PutMetricData call, statsd UDP
>    packet, or whatever)
> 
>  - a ceilometer notification handler or pollster provides a
>    Transformer corresponding to each of its Publishers, which
>    knows how to transform domain-specific raw data into the
>    relevant representation
> 
>  - the batching of transformer output is up to the Publisher
> 
>  - each Publisher has an independent frequency, so not all
>    Publishers are called to publish every measurement
> 
> So for example the compute.libvirt.DiskIOPollster would provide a
> transformer to map the raw disk stats reported by libvirt into
> DiskWriteOps, DiskWriteBytes, DiskReadOps & DiskReadBytes metric
> datapoints for CW, also indicating that this should be published
> very minute. This CWPublisher would handle batching these data
> as a single PutMetricData call.
> 
> The DiskIOPollster would also provide a second Transformer to
> mediate between the raw stats format and the ceilo Counter
> representation for metering.  Additionally, this second Publisher
> could be configured with a longer period (as befits metering data
> likely to be consumed by a billing engine).
> 
> Now the frequency aspect is a bit awkward. For notifications,
> the frequency of the "exists" style of usage audit is out of our
> hands, depending usually on the configuration of the emitter
> (e.g. the nova instance_usage_audit_period config). Whereas ceilo
> config direct controls only the frequency of polling. So one idea
> would be for a publisher's frequency only to impact on the periods
> under ceilo control - i.e. we'd accept and publish notifications
> whenever we get them, whereas the polling cycle would be determined
> by the greatest common divisor of all the configured publisher
> periods for each set of related pollsters (the config would probably
> have to be restricted so to ensure a sane GCD).
> 
> Also, I'm assuming that an alternative publication route would
> generally be used when the target endpoint is not the ceilo collector
> & metering store ... i.e. non-standard publisher would most often
> be used in order to hive off the data to an alternative endpoint
> or store, such as a CW metrics store, or say to be dumped as RRD
> files to be consumed by graphite.
> 
> Thoughts?
> 
> Cheers,
> Eoghan