Forgot to copy the list<br><br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Luis Gervaso</b> <span dir="ltr"><<a href="mailto:luis@woorea.es">luis@woorea.es</a>></span><br>
Date: Wed, May 9, 2012 at 7:14 PM<br>Subject: Re: [Openstack] ceilometer (java implementation)<br>To: Doug Hellmann <<a href="mailto:doug.hellmann@dreamhost.com">doug.hellmann@dreamhost.com</a>><br><br><br>I asked for these fields in my previous email, they are not clear enough.<br>
<br>After analyze the problem, i ended that metrics can not be calculated per event. We need at least 2 events to calculate the duration (start/end) in some metrics. We can afford this on the counter processes, but it requires an update of the previous pesisted start event. <br>
<br>So as we can calculate this info everywhere, and i don't like updates on the counters logic i'm calculating the duration as part of agreggation/mediation process (this will be only a view of the raw data).<br>
<br>This way we maintain a very simple logic on the counters:<br><br>* counters listen for specific events<br>* the interpretation of raw data can be outside<div class="HOEnZb"><div class="h5"><br><br><br><br><div class="gmail_quote">
On Wed, May 9, 2012 at 7:01 PM, Doug Hellmann <span dir="ltr"><<a href="mailto:doug.hellmann@dreamhost.com" target="_blank">doug.hellmann@dreamhost.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br><br><div class="gmail_quote"><div>On Wed, May 9, 2012 at 12:46 PM, Luis Gervaso <span dir="ltr"><<a href="mailto:luis@woorea.es" target="_blank">luis@woorea.es</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
That's fantastic!<br><br>What I mind is that the counters only should gather event info.<br><br>Maybe multiple counters listening, collecting info about the system. But only one persist the event.<br></blockquote><div>
<br></div></div><div>That makes sense. It seems like we will need to split the event-based processing from the polling. The agent that runs on the compute node should poll libvirt (and any other services on that node) and generate counter messages using a "metering" exchange. A separate set of daemons running in a more central location will watch messages on the metering exchange and save them to the database. Those same processes can watch for notification messages and convert them directly to metering data to be written to the database. That saves the extra traffic from notification messages resulting in several extra metering messages.</div>
<div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>Other process agreggates the data and creates different views/reports/billing, this should be outside of openstack (since it depends on the business rules, not depends on the infrastructure)<br>
</blockquote><div><br></div></div><div>If I understand Nick's proposal for the API, we will have some minimal aggregation there but you are right that any other interpretation will probably need to be handled by a custom app.</div>
<div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>The agreggator, also should offer an api for polling or can be configured throught drivers to push the the 3rd<br>systems<br><br>Now, in my implementation i'm putting all the stuff on a directory but it can be as well:<br>
<br>- AMQP <br>- SQL / NoSQL<br><br>(the persistent layer shold be interchangeable)<br><br>So as response to your duration question, this think is out scope for the counter tasks.</blockquote><div><br></div></div><div>I think you misunderstood my question. The "duration" to which I referred was the field in the counter schema ("counter_duration"). Only the "exists" notification for instances includes enough information to calculate the duration. The "create" notification should result in a "0" duration, so that can be ignored. The "delete" notification should record the data since the end of the last cycle, but does not today (at least it does not seem to).</div>
<div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><br><br><br><br><div class="gmail_quote">On Wed, May 9, 2012 at 5:30 PM, Doug Hellmann <span dir="ltr"><<a href="mailto:doug.hellmann@dreamhost.com" target="_blank">doug.hellmann@dreamhost.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br><br><div class="gmail_quote"><div>On Tue, May 8, 2012 at 7:19 PM, Luis Gervaso <span dir="ltr"><<a href="mailto:luis@woorea.es" target="_blank">luis@woorea.es</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br><br>I have uploaded a toy version of ceilometer (java implementation).<br><br>It does implement the first two counters (instance : rabbitmq listener and cpu : polling from libvirt)<br><br>i need more clarification on the meaining:<br>
<br>counter_volume<br>counter_duration<br>counter_datetaime<br><br>I hope this helps to figure out how to agreggate these data.<br><br><a href="http://github.com/woorea/ceilometer-java" target="_blank">http://github.com/woorea/ceilometer-java</a></blockquote>
<div><br></div></div><div>Nice!</div><div><br></div><div>I have also been experimenting. I have some Python code in <a href="https://github.com/dhellmann/metering-prototype" target="_blank">https://github.com/dhellmann/metering-prototype</a> that listens for notifications related to instances (create, delete, exists) and converts them to counter output (see spy.py). There is also a pair of scripts for recording an event stream and playing it back (useful for testing, see recorder.py and player.py). I don't have any libvirt polling, yet, though.</div>
<div><br></div><div>In the course of looking at the notifications being generated for different scenarios, I discovered that the instance delete messages do not have any duration information right now. I don't know if that is a bug, or if the idea is to figure out the durations from looking at the most recent "exists" event. What do other people think?</div>
<span><font color="#888888">
<div><br></div><div>Doug</div></font></span></div>
</blockquote></div><br><br clear="all"><br></div></div><div><div>-- <br>-------------------------------------------<br>Luis Alberto Gervaso Martin<div>Woorea Solutions, S.L<br>CEO & CTO<br>mobile: <a href="tel:%28%2B34%29%20627983344" value="+34627983344" target="_blank">(+34) 627983344</a><br>
<a href="mailto:luis.gervaso@gmail.com" target="_blank">luis@</a><a href="http://woorea.es/" target="_blank">woorea.es</a></div>
<br>
</div></div></blockquote></div></div><br>
</blockquote></div><br><br clear="all"><br>-- <br>-------------------------------------------<br>Luis Alberto Gervaso Martin<div>Woorea Solutions, S.L<br>CEO & CTO<br>mobile: <a href="tel:%28%2B34%29%20627983344" value="+34627983344" target="_blank">(+34) 627983344</a><br>
<a href="mailto:luis.gervaso@gmail.com" target="_blank">luis@</a><a href="http://woorea.es/" target="_blank">woorea.es</a></div>
<br>
</div></div></div><br><br clear="all"><br>-- <br>-------------------------------------------<br>Luis Alberto Gervaso Martin<div>Woorea Solutions, S.L<br>CEO & CTO<br>mobile: (+34) 627983344<br><a href="mailto:luis.gervaso@gmail.com" target="_blank">luis@</a><a href="http://woorea.es/" target="_blank">woorea.es</a></div>
<br>