Open Stack

Hi, folks.

I was interested in the problem with storing of samples, that contain
complex resource_metadata, in MongoDB database [1].

If data is a dict that has a  key(s) with dots (i.e. .), dollar signs (i.e.
$), or null characters,
it wouldn't be stored. It is happened because these characters are
restricted to use in fields name in MongoDB [2], but so far there is no any
verification of the metadata in ceilometers mongodb driver, as a result we
will lose data.

Solution of this problem seemed to be rather simple, before storing data we
check keys in resourse_metadata, if it is a dict, and "quote" keys with
restricted characters in a similar way, as it was done in a change request
of redesign separators in columns in HBase [2]. After that store metering
data.

But other unexpected difficulties appear on the step of getting data. To
get stored data we constructs a meta query, and structure of that query was
chosen identical to initial query in MongoDB. So dots is used as a
separator for three nods of stored data.

Ex. If it is needed to check value in field "Foo"

{metadata:
                    { Zoo:
                               {Foo: ''value"}}}

query would be: "metadata.Zoo.Foo"

We don't know how deep is dict in metadata, so it is impossible to propose
any correct parsing of query, to "quote" field names contain dots.

I see two way for improvements. First is rather complex and based of
redesign structure of the metadata query in ceilometer. Don't know if it is
ever possible.

And second is based on removing from the samples "bad" resource_metadata.
In this case we also lose metadata,  but save other metering data. Of
course queries for not saved metadata will return nothing, so it is not
complete solution, but some kind of the hook.

What do you think about that?
Any thoughts and propositions are kindly welcome.

[1] https://bugs.launchpad.net/mos/+bug/1360240
[2] http://docs.mongodb.org/manual/reference/limits/
[3] https://review.openstack.org/#/c/106376/

-- Igor Degtiarov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140829/62b81890/attachment.html>