Open Stack

On 01/30/2013 11:12 AM, Julien Danjou wrote:
> On Wed, Jan 30 2013, Sandy Walsh wrote:
> 
>> Actually ... what if the json payload of the raw event was stored as
>> metadata on a Meter?
> 
> What do you have in mind: storing the whole payload, or storing
> individual fields of the payload? Both looks different to me.

The raw notification is just a python dict anyway, so we could just
store it in a memo field (due to potentially large size). But there is a
*ton* of juicy stuff in there that isn't extracted in CM currently.

On StackTach we found many cases where the data we needed wasn't
extracted, but since we had the raw notification we were able to go back
and extract it. Otherwise, we would have had to capture it and then wait
for weeks until we had a suitable sample size.

> Currently we do store a lot of elements from the payload, but not all,
> because we don't have a use case for a lot, like the request_id.

Actually I only see about a half dozen fields being pulled from 3 events
in nova.

If ceilometer is going to be used as a debugging tool (certainly that's
under the Monitoring umbrella), request_id is critical.

> If your need is to store more components from the original payload into
> the metadata, that sounds like something possible.

Cool ... the divide isn't that great now :)

>> Is that permitted in Ceilometer?
> 
> Yes, it is. You could store that as a string in the key/value of a
> metadata.
> Usefulness and worthiness of this feature is still discutable.

I think that's something that should be decided by the deployment.

>> If so, this gets us back to a bigger and more interesting question:
>>
>> If we go with Metadata table + a Meter table we would have to index both
>> the Key and Value columns of the Metadata table.
>>
>> For example:
>>
>> Some metadata might be:
>>
>> "event_name" : "compute.instance.run_instance" and
>> "request_id" : "xxxx-xxxx-xxxx-xxxx"
>>
>> And we would want to see all <Foo> meters for request_id = 'xxxx-" and
>> event_name = "...."
>>
>> Having something in Metadata other than primitive data types would
>> result in a storage nightmare.
> 
> Yes, that's why we ruled it out.
> 
>> Which also brings up the point about storing multiple types in the Value
>> column under SQL (no-sql is fine). It would have to be a tagged union /
>> variant record to say which value is being used:
>>
>> Metadata
>> + key : string
>> + data_type : enum
>> + integer_value : int
>> + float_value : float
>> + string_value : string
>> + datetime_value : datetime
>> + blob_value : memo
>> ...
>>
>> So, I guess we could apply indices only the meaningful data type columns.
> 
> Looks like an idea, indeed.

A good idea? A viable idea? An idea worth pursuing?

(that said, I have no idea how this is all going to shake out in a
sharded mongo deployment ... if it's even possible? Need to do some
investigation for sure.)

-S