Open Stack

On Thu, Nov 22 2012, Eoghan Glynn wrote:

> The pollster or notification handler would basically just stuff all
> the available raw data into the sample kwargs. The transformers would
> then have to know which named args to expect and how to interpret the
> resource_obj.

I don't like the idea of having the transformer to guess which args is
going to get.

To me, the pollster or notifications handler should be responsible to
emit the maximum amount of "counters" it can from what it gets. And for
each counter, you would pass it through a transformer, mangling the
value to some over thing.

> From gerrit:
>
>   The point that I had envisaged for transformers was to factor
>   out the detailed per-measurement knowledge from the corresponding
>   publisher.
>
>   Take for example the stats related to disk I/O reported by the
>   hypervisor driver. Before these data can be pushed up to CloudWatch,
>   something has to know that the metric names are 'DiskWriteBytes',
>   'DiskReadBytes' etc., the namespace is 'AWS/EC2', the dimensions
>   include {'InstanceId': ID}, and the unit is Bytes.
>
>   So the idea was to avoid encoding all that knowledge in the CW
>   publisher, instead leaving the publisher simple and slow-changing
>   and unaffected by new metrics being added to the mix.
>
> Does that make sense at all?

Yes, it totally does.

The solution I'd imagine for that is to have the all pollster to emit
something equivalent (but probably simpler) to Counter like for example
in this case:

  { resource_name = 'disk.io',
    resource_id = 'instanceid.vda',
    user_id = 'qwerty789',
    tenant_id = 'abcdef123',
    value = 123456 }

So for CW you wouldn't transform the value, but the resource_name to
some other thing. This could be achieved via a transformer named
"RenameResourceName" which could be "configured" with a map:

  { "disk.io": "DiskWriteBytes",
     … }

So it's kind of generic and you can even use it to do some other stuff.

Does that make sense?

> Yeah, that's an idea. Wouldn't transformers have to be chained in that
> case? So for example the relevant CW transformer chain would be:
>
>   generic-transformer-calculating-cpu-util-from-cumulative-time -->
>     CW-specific-transformer-outputting-as-CPUUtilization-datapoint
>
> That would be fine, just wanted to call out a potential extra
> bit of complexity to capture in the pipeline config.

Yes, I think we'll need to chain them. Don't know how we can do that
easily in our configuration file.

OTOH, I imagine that most of the transformers configuration is going to
be pretty generic, like for CW. So we could define and provide a default
JSON file with all the default pipelining set-up correctly for most
counters and publishers.

-- 
Julien Danjou
// Free Software hacker & freelance
// http://julien.danjou.info
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121122/9c8be57e/attachment.pgp>