Open Stack

On 05/21/2013 03:56 PM, Doug Hellmann wrote:
> On Mon, May 20, 2013 at 3:38 PM, Sandy Walsh <sandy.walsh at rackspace.com
> <mailto:sandy.walsh at rackspace.com>> wrote:
>     Any time a new Metric is desired in Ceilometer a code change is
>     required (and all the downstream ramifications of that).
> 
> 
> FWIW, the notification plugins don't have to be in the ceilometer code
> base. That was one of the reasons for using entry points to load them.

True, just not something an installer would want to do.

>     Some of these meters are populated via Ceilometer's Pollsters and
>     some by processing Notifications, but the flow is the same:
> 
>     1. Data comes in from somewhere (an Event)
>     2. Parts of the Event are extracted and a Meter record is populated
>     3. The Meters are persisted
> 
>     A large portion of this Meter information is invariant (Meter name,
>     type, units of measure, etc). The rest are essentially
>     Traits/Metadata/Attributes on the Meter. Some attributes are
>     required, but most are optional.
> 
>     I'd like to see these Meters defined via some form of configuration
>     file.
> 
> 
> The meters derived from messages like notifications can be. You could
> resurrect the blueprint for doing this
> (https://blueprints.launchpad.net/ceilometer/+spec/config-driven-notification-monitoring).
> I think we dropped it when we decided to focus on the publishing side
> with multipublisher.

Yep, something like that exactly.

> 
> 
>     We should adopt the Oslo Notification as the incoming Event data
>     format and then offer a generic 
> 
> 
> The notification message format is only loosely defined, which is why we
> needed code to convert between it and something we could query
> consistently.We probably don't need hand-crafted code just to do those
> conversions, but at the time we built that piece we weren't sure that
> would always be the case (I originally thought we might have to look up
> some properties when the notification came in, at least until we were a
> mature enough project to convince the other projects to supplement their
> notifications with full metadata).

Yeah, this is the best we have so far
https://wiki.openstack.org/wiki/SystemUsageData ... but that would be
the next step in the process I think, to solidify the structures.

>  
> 
>     way to extract the fields from this structure and stuff them into a
>     Meter. Versioning for these events is underway already. Ceilometer
>     could ship with a set of .conf files for each OpenStack component
>     and we would only need to modify the Pollsters to spit out a
>     Notification-like format.
> 
> 
> I definitely like the idea of switching from an RPC message type to
> something that looks more like a notification. Again, the reason for the
> current implementation had to do with what was available at the time it
> was written.
>  

Understandable.

>     mapping:
>             'payload.image.type' -> 'image_type'
>             'payload.network.ip4 -> 'ip_address'
>             ...
> 
> 
> In a previous conversation on one of his patches, Julien and I agreed
> that it made sense to pass the metadata for a new meter through our
> existing dictionary-flattening function and store the results (instead
> of the nested dictionary we store now). If we applied that to the
> payload in notifications we receive, we could probably eliminate all
> cases where we need code to transform the data. A simple alias mapping
> like you describe could be optionally used to rename fields, but the
> vast majority of the fields would not need to be touched at all.
>  

Agreed. We could flatten or walk the data structure dynamically. Walking
might be faster since we're only going to pull out a small number of
fields and some of those notifications are huge. But, that's an
optimisation.

> 
> 
>     parser:
>             ceilometer.parser.dict_extractor # likely the default
> 
> 
>     It would probably even be possible to use a format that could
>     auto-generate the docs for these Meters.
> 
> 
> That would be a big win. I had been planning to do something like that
> by pulling data out of the plugins, but a config file would be just as easy.
> 
> 
>     This could dramatically reduce the amount of code in Ceilometer
>     today and greatly increase the ability for installers to customize
>     it without code changes.
> 
>     Thoughts?
> 
> 
> As far as implementation, if we keep the plugin system we have and add
> the new configuration-based system, then we can migrate to it slowly and
> maintain backwards compatibility (as well as continuing to provide a way
> for custom processing beyond the simple field renames).

Absolutely. Perhaps a factory that produces Meter objects. One factory
can pull from stevedore and one can pull from the config file. Both can
co-exist. The plug-in would be for the factory(s) to use.

> Doug
>  

Thanks!

> 
>     -S
> 
>     _______________________________________________
>     OpenStack-dev mailing list
>     OpenStack-dev at lists.openstack.org
>     <mailto:OpenStack-dev at lists.openstack.org>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>