<div dir="ltr"><div>Hi!<br><br>After some local discussions with Dmitiy Uklov we have found solution.<br><br>To solve the problem with dots inside keys in metadata dictionary we propose to reconstruct it<br>before storing in MongoDB.<br><br></div><div>Ex.<br><br></div><div>If we get metadata: {'a.s.d': 'v'}<br><br>it could be stored in MongoDB after a reconstruction<br><br></div><div>{'a': {'s': {'d': 'v'}}} <br><br>After storing in MongoDB value 'v' easily could be found with the standard query 'metadata.a.s.d'='v'.<br></div><div>Keys that start with '$' are "quoted" with the quote function from urllib.parse<br><br></div><div>I have proposed change request with fix: <a href="https://review.openstack.org/121003">https://review.openstack.org/121003</a><br></div><div><br></div><div><div><div>Cheers,<br></div><div><div><div class="gmail_extra"><div><div dir="ltr">-- Igor D.<br><br><br></div></div>
<br><div class="gmail_quote">On Mon, Sep 8, 2014 at 5:02 PM, Igor Degtiarov <span dir="ltr"><<a href="mailto:idegtiarov@mirantis.com" target="_blank">idegtiarov@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><span>On Thu, Sep 4, 2014 at 1:16 PM, Nadya Privalova <span dir="ltr"><<a href="mailto:nprivalova@mirantis.com" target="_blank">nprivalova@mirantis.com</a>></span> wrote:<br></span><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">IMHO it's ok and even very natural to expect "escaped" query from users. <div>e.g, we store the following structure </div><div><br></div><div><div style="font-family:arial,sans-serif;font-size:13px">
{metadata: <br> { Zoo: <br></div><span style="font-family:arial,sans-serif;font-size:13px"> {Foo.Boo</span><span style="font-family:arial,sans-serif;font-size:13px">: ''value"}}}</span></div></div></blockquote><div><br></div></span><div> Yep but such structure couldn't be stored in MongoDB without exchanging dot in "Foo.Boo"<br></div><span><div><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">and query should be "</span><span style="font-family:arial,sans-serif;font-size:13px">metadata.Zoo.Foo\.Boo" .</span></div></div></blockquote><div><br></div></span><div>That could be a good solution, but it is needed only if MongoDB is chosen as a backend. So the question is <br>should we specify query only for MongoDB or change queries for all backends? <br></div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">
<div><span style="font-family:arial,sans-serif;font-size:13px">In this case it's not necessary to know deep of tree.</span></div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px">Thanks,</span></div><div><span style="font-family:arial,sans-serif;font-size:13px">Nadya</span></div><div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div></div><div class="gmail_extra"><br></div></blockquote><div><br></div></span><div>Cheers,<br></div><div>Igor D.<br></div><div><div><div><br><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Fri, Aug 29, 2014 at 3:21 PM, Igor Degtiarov <span dir="ltr"><<a href="mailto:idegtiarov@mirantis.com" target="_blank">idegtiarov@mirantis.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div dir="ltr"><div><div><div><div><div><div><div><div><div><div><div><div><div><div>Hi, folks.<br><br></div>I was interested in the problem with storing of samples, that contain complex resource_metadata, in MongoDB database [1].<br>
<br></div>If data is a dict that has a key(s) with dots (i.e. <tt><span>.</span></tt>), dollar signs
(i.e. <tt><span>$</span></tt>), or null characters,<br></div>it wouldn't be stored. It is happened because these characters are restricted to use in fields name in MongoDB [2], but so far there is no any verification of the metadata in ceilometers mongodb driver, as a result we will lose data.<br>
<br></div>Solution of this problem seemed to be rather simple, before storing data we check keys in resourse_metadata, if it is a dict, and "quote" keys with restricted characters in a similar way, as it was done in a change request of redesign separators in columns in HBase [2]. After that store metering data.<br>
<br></div>But other unexpected difficulties appear on the step of getting data. To get stored data we constructs a meta query, and structure of that query was chosen identical to initial query in MongoDB. So dots is used as a separator for three nods of stored data.<br>
<br></div>Ex. If it is needed to check value in field "Foo"<br><br></div>{metadata: <br> { Zoo: <br></div> {Foo: ''value"}}}<br><br></div>query would be: "metadata.Zoo.Foo"<br>
<br></div>We don't know how deep is dict in metadata, so it is impossible to propose any correct parsing of query, to "quote" field names contain dots.<br><br></div>I see two way for improvements. First is rather complex and based of redesign structure of the metadata query in ceilometer. Don't know if it is ever possible.<br>
<br></div>And second is based on removing from the samples "bad" resource_metadata. In this case we also lose metadata, but save other metering data. Of course queries for not saved metadata will return nothing, so it is not complete solution, but some kind of the hook.<br>
<br></div>What do you think about that?<br></div>Any thoughts and propositions are kindly welcome.<br><div><div><div><div><div><div><div><div><div><div><br>[1] <a href="https://bugs.launchpad.net/mos/+bug/1360240" target="_blank">https://bugs.launchpad.net/mos/+bug/1360240</a><br>
[2] <a href="http://docs.mongodb.org/manual/reference/limits/" target="_blank">http://docs.mongodb.org/manual/reference/limits/</a><br>[3] <a href="https://review.openstack.org/#/c/106376/" target="_blank">https://review.openstack.org/#/c/106376/</a><span><font color="#888888"><br>
<div> <br><div><div><div><div><div><div dir="ltr">-- Igor Degtiarov<br></div></div>
</div></div></div></div></div></font></span></div></div></div></div></div></div></div></div></div></div></div>
<br></div></div>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div></div></div></div></div></div>