Open Stack

> > The pollster or notification handler would basically just
> > stuff all the available raw data into the sample kwargs. The
> > transformers would then have to know which named args to
> > expect and how to interpret the resource_obj.
> 
> I don't like the idea of having the transformer to guess which args
> is going to get.

Well I was thinking that the transformer would be written
with the inbuilt knowledge of what the corresponding pollster
is going to emit (as opposed to guessing).

> To me, the pollster or notifications handler should be
> responsible to emit the maximum amount of "counters" it can
> from what it gets. And for each counter, you would pass it
> through a transformer, mangling the value to some over thing.

Isn't the Counter just a tuple of name-value pairs?

I'm not sure how it would help to have the raw data passed via
a tuple as compared to kwargs?

> The solution I'd imagine for that is to have the all pollster to emit
> something equivalent (but probably simpler) to Counter like for
> example
> in this case:
> 
>   { resource_name = 'disk.io',
>     resource_id = 'instanceid.vda',
>     user_id = 'qwerty789',
>     tenant_id = 'abcdef123',
>     value = 123456 }
> 
> So for CW you wouldn't transform the value, but the resource_name to
> some other thing. This could be achieved via a transformer named
> "RenameResourceName" which could be "configured" with a map:
> 
>   { "disk.io": "DiskWriteBytes",
>      … }
> 
> So it's kind of generic and you can even use it to do some other
> stuff.
> 
> Does that make sense?

Interesting thought. I think the generic renaming approach could work
for the simplest cases, but say the CW transformer also had to construct
a few different dimensions for the metric?

For example for an instance metric, it would set the ImageId dimension
to the image UUID, the InstanceId dimension to the instance UUID, the
AutoScalingGroupName dimension to some other value taken from the instance
metadata.

In that case, the logic in the CW transformer would probably have to
be hand-crafted instead of being generic.

> > Yeah, that's an idea. Wouldn't transformers have to be chained in
> > that
> > case? So for example the relevant CW transformer chain would be:
> >
> >   generic-transformer-calculating-cpu-util-from-cumulative-time -->
> >     CW-specific-transformer-outputting-as-CPUUtilization-datapoint
> >
> > That would be fine, just wanted to call out a potential extra
> > bit of complexity to capture in the pipeline config.
> 
> Yes, I think we'll need to chain them. Don't know how we can do that
> easily in our configuration file.

Yeah, the common cfg format has only limited support for collection
types.

> OTOH, I imagine that most of the transformers configuration is
> going to be pretty generic, like for CW. So we could define and
> provide a default JSON file with all the default pipelining
> set-up correctly for most counters and publishers.

Yep, a JSON file might be better option here than the common cfg
format.

Cheers,
Eoghan