<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
On 05/14/2012 04:15 PM, Doug Hellmann wrote:
<blockquote
cite="mid:CADb+p3Q_91BJ_S_A-rWi=haWUYJoeTL=q3sJbpjUXwq6DmQ5tg@mail.gmail.com"
type="cite"><br>
<br>
<div class="gmail_quote">On Fri, May 11, 2012 at 3:55 PM, Loic
Dachary <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:loic@enovance.com" target="_blank">loic@enovance.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> - The interesting metadata for a resource may depend on
the type of<br>
> resource. Do we need separate "tables" for that or can we
normalize<br>
> somehow?<br>
> - How do we map a resource to the correct version of its
metadata at<br>
> any given time? Timestamps seem brittle.<br>
> - Do we need to reflect the metadata in the aggregation
API?<br>
><br>
Hi,<br>
<br>
I started a new thread for the "metadata" topic. I suspect it
deserves it. Although I was reluctant to acknowledge that the
metadate should be stored by the metering, yesterday's meeting
made me realize that it was mandatory. The compelling reason (
for me ;-) is that it would make it much more difficult to
implement a billing system if the metering does not provide a
simple way to extract metadata and display it in a human
readable way (or meaningfull to accountants ?) .<br>
<br>
I see two separate questions :<br>
<br>
a) how to store and query metadata ?<br>
b) what is the semantic of metadata for a given resource ?<br>
<br>
My hunch is that there will never be a definitive answer to b)
and that the best we can do is to provide a format and leave
the semantic to the documentation of the metering system,
explaining the metadata of a resource.<br>
<br>
Regarding the storage of the metadata, the metering could
listen / poll events creating / updating / deleting a given
resource and store a history log indexed by the resource id.
Something like:<br>
<br>
{ meter_type: TTT,<br>
resource_id: RRR,<br>
metadata: [{ version: VVVV,<br>
timestamp: TIME1,<br>
payload: PAYLOAD1 },<br>
{ version: VVVV,<br>
timestamp: TIME3,<br>
payload: PAYLOAD2 }]<br>
}<br>
<br>
With PPP being the resource dependant metadata that depends on
the type of the resource. And the metadata array being an
ordered list of the successive states of the resource over
time. The VVV version accounting for changes in the format of
the payload.<br>
<br>
The query would be :<br>
<br>
GET
/resource/<meter_type>/<resource_id>/<TIME2><br>
<br>
and it would return PAYLOAD1 if TIME2 is in the range
[TIME1,TIME3[<br>
<br>
I'm not sure why you think "timestamp is brittle". Maybe I'm
missing something.<br>
</blockquote>
<div><br>
</div>
<div>Each set of metering data will need to be associated with
the appropriate metadata from the resource at the time the
metering information was collected. The rate of change of
metadata and metering events are different, though, so the
timestamps of the metadata records are unlikely to match
exactly with the values in the metering records. Depending on
the clock resolution, it would be possible to have metadata
changes and meter data with the same timestamp, resulting in
an incorrect association.</div>
</div>
</blockquote>
Indeed, good point.<br>
<blockquote
cite="mid:CADb+p3Q_91BJ_S_A-rWi=haWUYJoeTL=q3sJbpjUXwq6DmQ5tg@mail.gmail.com"
type="cite">
<div class="gmail_quote">
<div><br>
</div>
<div>We can work around that by maintaining proper foreign key
references using the metadata version field as you describe in
the schema above (so the resource id and metadata version
value point to the correct metadata record). It will make
recording the metering data less efficient because we will
need to determine the current version for the resource
metadata, but we can optimize that eventually through indexes
and caching.</div>
<div><br>
</div>
<div>Aggregation will also need to take the metadata version
into account, so everywhere in the list of queries we say "by
resource_id" we need to change that to "by resource_id and
version".</div>
</div>
</blockquote>
I added the idea of a format version for when the payload format
changes and tried to write down a description of the metadata
storage matching this thread in the wiki.<br>
<br>
<a class="moz-txt-link-freetext" href="http://wiki.openstack.org/EfficientMetering?action=diff&rev2=80&rev1=78">http://wiki.openstack.org/EfficientMetering?action=diff&rev2=80&rev1=78</a><br>
<br>
What do you think ?<br>
<br>
<pre class="moz-signature" cols="3000">--
Loïc Dachary Chief Research Officer
// eNovance labs <a class="moz-txt-link-freetext" href="http://labs.enovance.com">http://labs.enovance.com</a>
// ✉ <a class="moz-txt-link-abbreviated" href="mailto:loic@enovance.com">loic@enovance.com</a> ☎ +33 1 49 70 99 82
</pre>
</body>
</html>