Open Stack

Hi!

After some local discussions with Dmitiy Uklov we have found solution.

To solve the problem with dots inside keys in metadata dictionary we
propose to reconstruct it
before storing in MongoDB.

Ex.

If we get metadata: {'a.s.d': 'v'}

it could be stored in MongoDB after a reconstruction

{'a': {'s': {'d': 'v'}}}

After storing in MongoDB value 'v' easily could be found with the standard
query 'metadata.a.s.d'='v'.
Keys that start with '$' are "quoted" with the quote function from
urllib.parse

I have proposed change request with fix:
https://review.openstack.org/121003

Cheers,
-- Igor D.



On Mon, Sep 8, 2014 at 5:02 PM, Igor Degtiarov <idegtiarov at mirantis.com>
wrote:

> On Thu, Sep 4, 2014 at 1:16 PM, Nadya Privalova <nprivalova at mirantis.com>
> wrote:
>
>> IMHO it's ok and even very natural to expect "escaped" query from users.
>> e.g, we store the following structure
>>
>> {metadata:
>>                     { Zoo:
>>                                {Foo.Boo: ''value"}}}
>>
>
>  Yep but such structure couldn't be stored in MongoDB without exchanging
> dot in "Foo.Boo"
>
>
>
>>
>> and query should be "metadata.Zoo.Foo\.Boo" .
>>
>
> That could be a good  solution, but it is needed only if MongoDB is chosen
> as a backend. So the question is
> should we specify query only for MongoDB or change queries for all
> backends?
>
>
>> In this case it's not necessary to know deep of tree.
>>
>> Thanks,
>> Nadya
>>
>>
>>
> Cheers,
> Igor D.
>
>
>
>
>>
>> On Fri, Aug 29, 2014 at 3:21 PM, Igor Degtiarov <idegtiarov at mirantis.com>
>> wrote:
>>
>>> Hi, folks.
>>>
>>> I was interested in the problem with storing of samples, that contain
>>> complex resource_metadata, in MongoDB database [1].
>>>
>>> If data is a dict that has a  key(s) with dots (i.e. .), dollar signs
>>> (i.e. $), or null characters,
>>> it wouldn't be stored. It is happened because these characters are
>>> restricted to use in fields name in MongoDB [2], but so far there is no any
>>> verification of the metadata in ceilometers mongodb driver, as a result we
>>> will lose data.
>>>
>>> Solution of this problem seemed to be rather simple, before storing data
>>> we check keys in resourse_metadata, if it is a dict, and "quote" keys with
>>> restricted characters in a similar way, as it was done in a change request
>>> of redesign separators in columns in HBase [2]. After that store metering
>>> data.
>>>
>>> But other unexpected difficulties appear on the step of getting data. To
>>> get stored data we constructs a meta query, and structure of that query was
>>> chosen identical to initial query in MongoDB. So dots is used as a
>>> separator for three nods of stored data.
>>>
>>> Ex. If it is needed to check value in field "Foo"
>>>
>>> {metadata:
>>>                     { Zoo:
>>>                                {Foo: ''value"}}}
>>>
>>> query would be: "metadata.Zoo.Foo"
>>>
>>> We don't know how deep is dict in metadata, so it is impossible to
>>> propose any correct parsing of query, to "quote" field names contain dots.
>>>
>>> I see two way for improvements. First is rather complex and based of
>>> redesign structure of the metadata query in ceilometer. Don't know if it is
>>> ever possible.
>>>
>>> And second is based on removing from the samples "bad"
>>> resource_metadata. In this case we also lose metadata,  but save other
>>> metering data. Of course queries for not saved metadata will return
>>> nothing, so it is not complete solution, but some kind of the hook.
>>>
>>> What do you think about that?
>>> Any thoughts and propositions are kindly welcome.
>>>
>>> [1] https://bugs.launchpad.net/mos/+bug/1360240
>>> [2] http://docs.mongodb.org/manual/reference/limits/
>>> [3] https://review.openstack.org/#/c/106376/
>>>
>>> -- Igor Degtiarov
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140912/1264d818/attachment-0001.html>