Open Stack

---------- Forwarded message ----------
From: Luis Gervaso <luis at woorea.es>
Date: Wed, May 9, 2012 at 7:59 PM
Subject: Re: [Openstack] ceilometer (java implementation)
To: Doug Hellmann <doug.hellmann at dreamhost.com>


Hi,

For now i'm using sql and it works perfectly for simple use cases.

I am pretty sure we need to integrate with a 3rd party. Zoura was my first
chooice, i tried to contact with them but no response :(. So i have started
with a simple solution, own implemented with SQL as starting point for
aggregation and billing.

Attached an image of my vision.

Comments are welcome!






On Wed, May 9, 2012 at 7:31 PM, Doug Hellmann
<doug.hellmann at dreamhost.com>wrote:

>
>
> On Wed, May 9, 2012 at 1:14 PM, Luis Gervaso <luis at woorea.es> wrote:
>
>> I asked for these fields in my previous email, they are not clear enough.
>>
>> After analyze the problem, i ended that metrics can not be calculated per
>> event. We need at least 2 events to calculate the duration (start/end) in
>> some metrics. We can afford this on the counter processes, but it requires
>> an update of the previous pesisted start event.
>>
>
> The exists notifications are generated periodically and the code that does
> that knows how often they are being generated, so it includes a duration.
>
>
>>
>> So as we can calculate this info everywhere, and i don't like updates on
>> the counters logic i'm calculating the duration as part of
>> agreggation/mediation process (this will be only a view of the raw data).
>>
>> This way we maintain a very simple logic on the counters:
>>
>> * counters listen for specific events
>> * the interpretation of raw data can be outside
>
>
> What are you using as a start point for the aggregation in a case where an
> instance has been alive for more than a billing cycle?
>
>
>>
>>
>>
>>
>> On Wed, May 9, 2012 at 7:01 PM, Doug Hellmann <
>> doug.hellmann at dreamhost.com> wrote:
>>
>>>
>>>
>>> On Wed, May 9, 2012 at 12:46 PM, Luis Gervaso <luis at woorea.es> wrote:
>>>
>>>> That's fantastic!
>>>>
>>>> What I mind is that the counters only should gather event info.
>>>>
>>>> Maybe multiple counters listening, collecting info about the system.
>>>> But only one persist the event.
>>>>
>>>
>>> That makes sense. It seems like we will need to split the event-based
>>> processing from the polling. The agent that runs on the compute node should
>>> poll libvirt (and any other services on that node) and generate counter
>>> messages using a "metering" exchange. A separate set of daemons running in
>>> a more central location will watch messages on the metering exchange and
>>> save them to the database. Those same processes can watch for notification
>>> messages and convert them directly to metering data to be written to the
>>> database. That saves the extra traffic from notification messages resulting
>>> in several extra metering messages.
>>>
>>>
>>>>
>>>> Other process agreggates the data and creates different
>>>> views/reports/billing, this should be outside of openstack (since it
>>>> depends on the business rules, not depends on the infrastructure)
>>>>
>>>
>>> If I understand Nick's proposal for the API, we will have some minimal
>>> aggregation there but you are right that any other interpretation will
>>> probably need to be handled by a custom app.
>>>
>>>
>>>>
>>>> The agreggator, also should offer an api for polling or can be
>>>> configured throught drivers to push the the 3rd
>>>> systems
>>>>
>>>> Now, in my implementation i'm putting all the stuff on a directory but
>>>> it can be as well:
>>>>
>>>> - AMQP
>>>> - SQL / NoSQL
>>>>
>>>> (the persistent layer shold be interchangeable)
>>>>
>>>> So as response to your duration question, this think is out scope for
>>>> the counter tasks.
>>>
>>>
>>> I think you misunderstood my question. The "duration" to which I
>>> referred was the field in the counter schema ("counter_duration"). Only the
>>> "exists" notification for instances includes enough information to
>>> calculate the duration. The "create" notification should result in a "0"
>>> duration, so that can be ignored. The "delete" notification should record
>>> the data since the end of the last cycle, but does not today (at least it
>>> does not seem to).
>>>
>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, May 9, 2012 at 5:30 PM, Doug Hellmann <
>>>> doug.hellmann at dreamhost.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, May 8, 2012 at 7:19 PM, Luis Gervaso <luis at woorea.es> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have uploaded a toy version of ceilometer (java implementation).
>>>>>>
>>>>>> It does implement the first two counters (instance : rabbitmq
>>>>>> listener and cpu : polling from libvirt)
>>>>>>
>>>>>> i need more clarification on the meaining:
>>>>>>
>>>>>> counter_volume
>>>>>> counter_duration
>>>>>> counter_datetaime
>>>>>>
>>>>>> I hope this helps to figure out how to agreggate these data.
>>>>>>
>>>>>> http://github.com/woorea/ceilometer-java
>>>>>
>>>>>
>>>>> Nice!
>>>>>
>>>>> I have also been experimenting. I have some Python code in
>>>>> https://github.com/dhellmann/metering-prototype that listens for
>>>>> notifications related to instances (create, delete, exists) and converts
>>>>> them to counter output (see spy.py). There is also a pair of scripts for
>>>>> recording an event stream and playing it back (useful for testing, see
>>>>> recorder.py and player.py). I don't have any libvirt polling, yet, though.
>>>>>
>>>>> In the course of looking at the notifications being generated for
>>>>> different scenarios, I discovered that the instance delete messages do not
>>>>> have any duration information right now. I don't know if that is a bug, or
>>>>> if the idea is to figure out the durations from looking at the most recent
>>>>> "exists" event. What do other people think?
>>>>>
>>>>> Doug
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> -------------------------------------------
>>>> Luis Alberto Gervaso Martin
>>>> Woorea Solutions, S.L
>>>> CEO & CTO
>>>> mobile: (+34) 627983344
>>>> luis@ <luis.gervaso at gmail.com>woorea.es
>>>>
>>>>
>>>
>>
>>
>> --
>> -------------------------------------------
>> Luis Alberto Gervaso Martin
>> Woorea Solutions, S.L
>> CEO & CTO
>> mobile: (+34) 627983344
>> luis@ <luis.gervaso at gmail.com>woorea.es
>>
>>
>


-- 
-------------------------------------------
Luis Alberto Gervaso Martin
Woorea Solutions, S.L
CEO & CTO
mobile: (+34) 627983344
luis@ <luis.gervaso at gmail.com>woorea.es




-- 
-------------------------------------------
Luis Alberto Gervaso Martin
Woorea Solutions, S.L
CEO & CTO
mobile: (+34) 627983344
luis@ <luis.gervaso at gmail.com>woorea.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20120509/e1c087ba/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ceilometer-diagram.001.jpg
Type: image/jpeg
Size: 79627 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20120509/e1c087ba/attachment.jpg>