<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, May 20, 2013 at 3:38 PM, Sandy Walsh <span dir="ltr"><<a href="mailto:sandy.walsh@rackspace.com" target="_blank">sandy.walsh@rackspace.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hey!<br>
<br>
Currently all the Meters in Ceilometer are defined programmatically. That is, there is specific code in the product to define Meters for Nova, Glance, Quantum, etc.<br>
<br>
These are all outlined here:<br>
<a href="http://docs.openstack.org/developer/ceilometer/measurements.html" target="_blank">http://docs.openstack.org/developer/ceilometer/measurements.html</a><br>
<br>
Any time a new Metric is desired in Ceilometer a code change is required (and all the downstream ramifications of that).<br></blockquote><div><br></div><div style>FWIW, the notification plugins don't have to be in the ceilometer code base. That was one of the reasons for using entry points to load them.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Some of these meters are populated via Ceilometer's Pollsters and some by processing Notifications, but the flow is the same:<br>
<br>
1. Data comes in from somewhere (an Event)<br>
2. Parts of the Event are extracted and a Meter record is populated<br>
3. The Meters are persisted<br>
<br>
A large portion of this Meter information is invariant (Meter name, type, units of measure, etc). The rest are essentially Traits/Metadata/Attributes on the Meter. Some attributes are required, but most are optional.<br>
<br>
I'd like to see these Meters defined via some form of configuration file.<br></blockquote><div><br></div><div style>The meters derived from messages like notifications can be. You could resurrect the blueprint for doing this (<a href="https://blueprints.launchpad.net/ceilometer/+spec/config-driven-notification-monitoring">https://blueprints.launchpad.net/ceilometer/+spec/config-driven-notification-monitoring</a>). I think we dropped it when we decided to focus on the publishing side with multipublisher.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
We should adopt the Oslo Notification as the incoming Event data format and then offer a generic </blockquote><div><br></div><div style>The notification message format is only loosely defined, which is why we needed code to convert between it and something we could query consistently.We probably don't need hand-crafted code just to do those conversions, but at the time we built that piece we weren't sure that would always be the case (I originally thought we might have to look up some properties when the notification came in, at least until we were a mature enough project to convince the other projects to supplement their notifications with full metadata).</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">way to extract the fields from this structure and stuff them into a Meter. Versioning for these events is underway already. Ceilometer could ship with a set of .conf files for each OpenStack component and we would only need to modify the Pollsters to spit out a Notification-like format.<br>
</blockquote><div><br></div><div style>I definitely like the idea of switching from an RPC message type to something that looks more like a notification. Again, the reason for the current implementation had to do with what was available at the time it was written.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
The configuration format could optionally point to a piece of code if complex parsing is required, but I'm sure 90% of the cases could be handled with simple drivers.<br></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Let's say an event looks like this:<br>
<br>
Event<br>
{<br>
'name' : 'Foo',<br>
'generated': <datetime>,<br>
'host': 'host_29',<br>
'service': 'api'<br>
'payload' : { ... specific stuff ... }<br>
...<br>
}<br>
<br>
<br>
Our nova_meters.conf might be something like:<br>
<br>
event:<br>
'nova.compute.run_instance.end'<br>
<br>
meter: compute.instances<br>
type: counter<br>
units: num<br>
<br>
mapping:<br>
'payload.image.type' -> 'image_type'<br>
'payload.network.ip4 -> 'ip_address'<br>
...<br></blockquote><div><br></div><div><div>In a previous conversation on one of his patches, Julien and I agreed that it made sense to pass the metadata for a new meter through our existing dictionary-flattening function and store the results (instead of the nested dictionary we store now). If we applied that to the payload in notifications we receive, we could probably eliminate all cases where we need code to transform the data. A simple alias mapping like you describe could be optionally used to rename fields, but the vast majority of the fields would not need to be touched at all.</div>
<div> </div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
parser:<br>
ceilometer.parser.dict_extractor # likely the default<br>
<br>
<br>
It would probably even be possible to use a format that could auto-generate the docs for these Meters.<br></blockquote><div><br></div><div style>That would be a big win. I had been planning to do something like that by pulling data out of the plugins, but a config file would be just as easy.</div>
<div style><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
This could dramatically reduce the amount of code in Ceilometer today and greatly increase the ability for installers to customize it without code changes.<br>
<br>
Thoughts?<br></blockquote><div><br></div><div style>As far as implementation, if we keep the plugin system we have and add the new configuration-based system, then we can migrate to it slowly and maintain backwards compatibility (as well as continuing to provide a way for custom processing beyond the simple field renames).</div>
<div style><br></div><div style>Doug</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
-S<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div><br></div></div>