Open Stack

----- Original Message -----
> Julien:
> 	I checked the IRC log yesterday and also patch from Angus
> 	(https://github.com/asalkeld/ceilometer/commit/c678ff8abd2a91c4404657748b9ed9b31c1541df
> 	), I'm a bit confused.
> 
> 	From Angus's patch and the discussion, IIUC, all data will be stored
> 	as meter in the storage, and the API handler will translate from
> 	meter to CW format (dimensions, Namespace, unit etc). This is a
> 	different to my understanding, so I want to get the big picture of
> 	monitor support and multiple publisher.
> 
> 	Firstly , some background clarification. I think currently when we
> 	talking about monitor, we are mostly talking about CW support, or
> 	CW-similar support, when we talking about multiple publisher, we
> 	are talking about CW publisher, at least in the near future (like
> 	in H range). Am I right?

Yes, the main initial use-case in mind for multi-publisher is the ability
to push directly to an "external" CloudWatch-like service via its public
PutMetricData API.

> 	Then, in my original understanding, in CW/monitor usage, the raw
> 	data collected from pollsters will be translated into CW format
> 	(MetricName, Namespace, Dimensions, Unit, Value) and we will
> 	provide the Metrics provided by AWS.

Yes, that is what I had envisaged also.

> 	Also I envision the CW data will be stored in the storage, with a
> 	dimensions table, and CW data in another table. Each
> 	(metricname+dimensions) combination will have an entry on the
> 	dimension table. A table for the alarm, and alarm table will be
> 	updated when new data added.

Yes, I would expect that it would involve some different data structures in
the store.

In fact it could well be a completely different store (e.g. a metric store
based on Cassandra, while the metering store defaults to Mongo).

>  	The aggregation will be achieved through passing dimensions as a
>  	filter parameter to the query.

Well you can think of CW aggregation as encompassing a slice over a
particular dimension (all instances with ImageId dimension = foo),
and/or as the application of a chosen statistic (sum, min, max,
sample count, average).

> 	With this envision, the translation from Counter to CW data will
> 	happen in publisher

In the transformer, not the publisher, I would have thought.

>       and we may need various type of transformer to
> 	achieve it, including a) Unit translation like from kb to byte, b)
> 	interval translation like from 10s to 60s,

I don't think we need interval translation. The metric is just pushed to
CW when it becomes available. The push doesn't involve specifying a period.
Instead the period (e.g. 60s) is specified when the metric is queried. If
the polling interval was set to 10s, then a GetMetricStatistics call with
period=60s would result in aggregation over the ~6 datapoints per period.

>       c) aggregation like
> 	summary all vnic data into one network in/out CW data, d) translate
> 	Counter format to CW data format, e) delta calculation like the CPU
> 	utilization. And we deserve the transformer mechanism to achieve
> 	them.

Yes, those would be some of the tasks envisaged for the transformer.
 
> 	However, if we will achieve CW data from API side (that also make
> 	sense and seems more flexible), then I don't think any transformer
> 	needed, except the CPU utilization, which has been implemented
> 	already. The interval in fact does not matter in such situation,
> 	except the interval of the pollster.

Well it depends on whether the CW service in integral to ceilo, or at
arms length. In the latter case, e.g. Synaps, the CW publisher would do
the metric push via the CW public API.

> 	Possibly I lost something important, or we will have more publisher
> 	to support in near future other than CW?

The idea was to also support publishing via statsd UDP packets, RRD files,
maybe even the librato metrics API, or whatever other conduit makes sense.

Cheers,
Eoghan