Open Stack

On 01/23/2013 10:40 PM, Lu, Lianhao wrote:
> Julien Danjou wrote on 2013-01-23:
>> On Wed, Jan 23 2013, Lu, Lianhao wrote:
>>
>>> Hi guys,
>>>
>>> If my understanding is correct, the current resource_metadata stored by the ceilometer collector could be any python object which can
>> be serialized to the JSON format, so the resource_metadata could be a dict containing values of dict/list/string, e.g.
>>
>>>
>>>   {"key1": [
>>>           { "key11": 11},
>>>           { "key12": 12}
>>>          ],
>>>    "key2": "v2"
>>>    "key3": {
>>>         "key31": 31
>>>        }
>>>   }
>>> Does the metaquery support complex conditions, like the form of
>>> metadata["key3"]["key31"] > 30 or { "key11": 11} in metadata["key1"],
>>> besides the simple form of metadata["key2"]=="v2"?
>>
>> Nop. The plan for now is to limit metadata querying to key = value where
>> value is of a simple type (string, int). I think it's a sane limitation,
>> and that will allow us to index more efficiently metadata.
>>
>>> Besides, supporting the metaquery in sqlalchemy is not easy as that in
>>> mongodb. The question is where should the query filter operation
>>> happen? We may do the metadata query filter operation on the
>>> sqlalchemy client side in python, which is comparably easy to
>>> implement and easy to be adapted to complex filter conditions, but is
>>> bad in performance because it needs to get all the records from
>>> backend DB to do that.
>>
>> We discussed this with Nick back then, and IIRC the idea is to add a
>> metadata table with colums 'meter_id', 'key' and 'value' and to index
>> those last fields.
>> If we were using PG I'd suggest to try arrays or PL/v8 to do JSON
>> filtering, but since we have to support MySQL that's not an option I
>> guess. D'oh.
> 
> I guess what you mentioned here is the table definition in 
> http://wiki.openstack.org/Ceilometer/blueprints/multi-dimensions.

This seems like a very inefficient schema requiring multiple sub-queries.

Other than the naming, is it really any different than the current
Metadata table when it comes to db performance?

I think a better approach would be to offer different Metric types
(extensible) which can control their own mapping to the db.

> 
> One thing concerns me is that it can only handle the first level key/pair
> in metadata. What about the other level keypair, e.g. {"key3": {"key31": 31}}?
> Should we flatten the metadata into {"key3.key31": 31} and store that into the
> metadata table? 
> 
> What's more worse is what if the metadata looks like the following? 
>   {"key1": [ { "key11": 11}, { "key12": 12} ] }
> 
> Or should we put some restriction on the metadata that it must be key/value
> dict, in which the value could only be sting/int/float or another dict abiding the same
> rules?

I'd be curious to see how the metadata table approach performs when you
are querying on multiple keys (like Event Name + Cell + Host + Request
ID, for example) with a large number of rows. Has anyone tried this?

-S



> -Lianhao
>>
>>> The other way is to do the metadata query filter operation on the
>>> backend DB side which should have better performance, but I don't know
>>> how to do that to support complex conditions other than the simple
>>> form of metadata["key2"]=="v2".
>>
>> You got the solution: we don't allow that. :)
>>
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>