Open Stack

On Thu, Sep 4, 2014 at 1:16 PM, Nadya Privalova <nprivalova at mirantis.com>
wrote:

> IMHO it's ok and even very natural to expect "escaped" query from users.
> e.g, we store the following structure
>
> {metadata:
>                     { Zoo:
>                                {Foo.Boo: ''value"}}}
>

 Yep but such structure couldn't be stored in MongoDB without exchanging
dot in "Foo.Boo"



>
> and query should be "metadata.Zoo.Foo\.Boo" .
>

That could be a good  solution, but it is needed only if MongoDB is chosen
as a backend. So the question is
should we specify query only for MongoDB or change queries for all
backends?


> In this case it's not necessary to know deep of tree.
>
> Thanks,
> Nadya
>
>
>
Cheers,
Igor D.




>
> On Fri, Aug 29, 2014 at 3:21 PM, Igor Degtiarov <idegtiarov at mirantis.com>
> wrote:
>
>> Hi, folks.
>>
>> I was interested in the problem with storing of samples, that contain
>> complex resource_metadata, in MongoDB database [1].
>>
>> If data is a dict that has a  key(s) with dots (i.e. .), dollar signs
>> (i.e. $), or null characters,
>> it wouldn't be stored. It is happened because these characters are
>> restricted to use in fields name in MongoDB [2], but so far there is no any
>> verification of the metadata in ceilometers mongodb driver, as a result we
>> will lose data.
>>
>> Solution of this problem seemed to be rather simple, before storing data
>> we check keys in resourse_metadata, if it is a dict, and "quote" keys with
>> restricted characters in a similar way, as it was done in a change request
>> of redesign separators in columns in HBase [2]. After that store metering
>> data.
>>
>> But other unexpected difficulties appear on the step of getting data. To
>> get stored data we constructs a meta query, and structure of that query was
>> chosen identical to initial query in MongoDB. So dots is used as a
>> separator for three nods of stored data.
>>
>> Ex. If it is needed to check value in field "Foo"
>>
>> {metadata:
>>                     { Zoo:
>>                                {Foo: ''value"}}}
>>
>> query would be: "metadata.Zoo.Foo"
>>
>> We don't know how deep is dict in metadata, so it is impossible to
>> propose any correct parsing of query, to "quote" field names contain dots.
>>
>> I see two way for improvements. First is rather complex and based of
>> redesign structure of the metadata query in ceilometer. Don't know if it is
>> ever possible.
>>
>> And second is based on removing from the samples "bad" resource_metadata.
>> In this case we also lose metadata,  but save other metering data. Of
>> course queries for not saved metadata will return nothing, so it is not
>> complete solution, but some kind of the hook.
>>
>> What do you think about that?
>> Any thoughts and propositions are kindly welcome.
>>
>> [1] https://bugs.launchpad.net/mos/+bug/1360240
>> [2] http://docs.mongodb.org/manual/reference/limits/
>> [3] https://review.openstack.org/#/c/106376/
>>
>> -- Igor Degtiarov
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140908/2d98a845/attachment.html>