Open Stack

On Mon, May 20, 2013 at 3:38 PM, Sandy Walsh <sandy.walsh at rackspace.com>wrote:

> Hey!
>
> Currently all the Meters in Ceilometer are defined programmatically. That
> is, there is specific code in the product to define Meters for Nova,
> Glance, Quantum, etc.
>
> These are all outlined here:
> http://docs.openstack.org/developer/ceilometer/measurements.html
>
> Any time a new Metric is desired in Ceilometer a code change is required
> (and all the downstream ramifications of that).
>

FWIW, the notification plugins don't have to be in the ceilometer code
base. That was one of the reasons for using entry points to load them.


>
> Some of these meters are populated via Ceilometer's Pollsters and some by
> processing Notifications, but the flow is the same:
>
> 1. Data comes in from somewhere (an Event)
> 2. Parts of the Event are extracted and a Meter record is populated
> 3. The Meters are persisted
>
> A large portion of this Meter information is invariant (Meter name, type,
> units of measure, etc). The rest are essentially Traits/Metadata/Attributes
> on the Meter. Some attributes are required, but most are optional.
>
> I'd like to see these Meters defined via some form of configuration file.
>

The meters derived from messages like notifications can be. You could
resurrect the blueprint for doing this (
https://blueprints.launchpad.net/ceilometer/+spec/config-driven-notification-monitoring).
I think we dropped it when we decided to focus on the publishing side with
multipublisher.


>
> We should adopt the Oslo Notification as the incoming Event data format
> and then offer a generic


The notification message format is only loosely defined, which is why we
needed code to convert between it and something we could query
consistently.We probably don't need hand-crafted code just to do those
conversions, but at the time we built that piece we weren't sure that would
always be the case (I originally thought we might have to look up some
properties when the notification came in, at least until we were a mature
enough project to convince the other projects to supplement their
notifications with full metadata).


> way to extract the fields from this structure and stuff them into a Meter.
> Versioning for these events is underway already. Ceilometer could ship with
> a set of .conf files for each OpenStack component and we would only need to
> modify the Pollsters to spit out a Notification-like format.
>

I definitely like the idea of switching from an RPC message type to
something that looks more like a notification. Again, the reason for the
current implementation had to do with what was available at the time it was
written.


>
> The configuration format could optionally point to a piece of code if
> complex parsing is required, but I'm sure 90% of the cases could be handled
> with simple drivers.
>

> Let's say an event looks like this:
>
> Event
> {
> 'name' : 'Foo',
> 'generated': <datetime>,
> 'host': 'host_29',
> 'service': 'api'
> 'payload' : { ... specific stuff ... }
> ...
> }
>
>
> Our nova_meters.conf might be something like:
>
> event:
>         'nova.compute.run_instance.end'
>
> meter: compute.instances
> type: counter
> units: num
>
> mapping:
>         'payload.image.type' -> 'image_type'
>         'payload.network.ip4 -> 'ip_address'
>         ...
>

In a previous conversation on one of his patches, Julien and I agreed that
it made sense to pass the metadata for a new meter through our existing
dictionary-flattening function and store the results (instead of the nested
dictionary we store now). If we applied that to the payload in
notifications we receive, we could probably eliminate all cases where we
need code to transform the data. A simple alias mapping like you describe
could be optionally used to rename fields, but the vast majority of the
fields would not need to be touched at all.


>
> parser:
>         ceilometer.parser.dict_extractor # likely the default
>
>
> It would probably even be possible to use a format that could
> auto-generate the docs for these Meters.
>

That would be a big win. I had been planning to do something like that by
pulling data out of the plugins, but a config file would be just as easy.


> This could dramatically reduce the amount of code in Ceilometer today and
> greatly increase the ability for installers to customize it without code
> changes.
>
> Thoughts?
>

As far as implementation, if we keep the plugin system we have and add the
new configuration-based system, then we can migrate to it slowly and
maintain backwards compatibility (as well as continuing to provide a way
for custom processing beyond the simple field renames).

Doug


>
> -S
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130521/73116084/attachment.html>