Open Stack

I take it that the instance manager doesn't generate any kind of heartbeat, so whatever monitoring/archiving service we do should internally poll the status over MQ?


-------------------------------------------------
Brian Schott, CTO
Nimbis Services, Inc.
brian.schott at nimbisservices.com
ph: 443-274-6064  fx: 443-274-6060



On Apr 24, 2012, at 2:10 PM, Luis Gervaso wrote:

> Probably an extra audit system is required. I'm searching for solutions in the IT market.
> 
> Regards
> 
> On Tue, Apr 24, 2012 at 6:00 PM, Loic Dachary <loic at enovance.com> wrote:
> On 04/24/2012 04:45 PM, Monsyne Dragon wrote:
>> 
>> 
>> On Apr 24, 2012, at 9:03 AM, Loic Dachary wrote:
>> 
>>> On 04/24/2012 03:06 PM, Monsyne Dragon wrote:
>>>> 
>>>> Yes,  we emit bandwidth (bytes in/out) on a per VIF basis from each instance The event has the somewhat generic name of 'compute.instance.exists'  and is emitted on an periodic basis, currently by a cronjob. 
>>>> Currently, we only populate bandwidth data from XenServer, but if the hook is implemented for  the kvm, etc drivers, it will be picked up automatically for them as well. 
>>>> 
>>>> Note that we could report other metrics similarly. 
>>> Hi,
>>> 
>>> Thanks for clarifying this. So you're suggesting that the metering agent should collect this data from the nova queue instead of extracting it from the system (interface, disk stats etc.) ? And for other openstack components ( as Nick Barcet suggests below ) the metering agent will have to find another way. Or do you have something else in mind ?
>> 
>> If it's something we have access to, we should emit it in those usage events.  As far as the other components, glance is already using the same notification system.  (there was a thread awhile back about putting it into openstack.common)  It would be nice to have all of the components using it.  
>> 
> Hi,
> 
> I don't see a section in http://wiki.openstack.org/SystemUsageData about making sure all messages related to a billable event are accounted for. I mean, for instance, what if the event that says an instance is deleted is lost ? How is the billing software supposed to cope with that ? If it checks the status of all VM on a regular basis to deal with this, how can it figure out when the missed event occured ?
> 
> It would be worth adding a short section about this in http://wiki.openstack.org/SystemUsageData . Or I can do it if you give me a hint.
> 
> Cheers
> 
>>> Cheers
>>> 
>>> On 04/24/2012 12:17 PM, Nick Barcet wrote:
>>>> 
>>>> On 04/23/2012 10:45 PM, Doug Hellmann wrote:
>>>>> > 
>>>>> > 
>>>>> > On Mon, Apr 23, 2012 at 4:14 PM, Brian Schott
>>>>> > <brian.schott at nimbisservices.com
>>>>> > <mailto:brian.schott at nimbisservices.com>> wrote:
>>>>> > 
>>>>> >     Doug,
>>>>> > 
>>>>> >     Do we mirror the table structure of nova, etc. and add
>>>>> >     created/modified columns? 
>>>>> > 
>>>>> > 
>>>>> >     Or do we flatten into an instance event record with everything?  
>>>>> > 
>>>>> > 
>>>>> > I lean towards flattening the data as it is recorded and making a second
>>>>> > pass during the bill calculation. You need to record instance
>>>>> > modifications separately from the creation, especially if the
>>>>> > modification changes the billing rate. So you might have records for:
>>>>> > 
>>>>> > created instance, with UUID, name, size, timestamp, ownership
>>>>> > information, etc.
>>>>> > resized instance, with UUID, name, new size, timestamp, ownership
>>>>> > information, etc.
>>>>> > deleted instance, with UUID, name, size, timestamp, ownership
>>>>> > information, etc.
>>>>> > 
>>>>> > Maybe some of those values don't need to be reported in some cases, but
>>>>> > if you record a complete picture of the state of the instance then the
>>>>> > code that aggregates the event records to produce billing information
>>>>> > can use it to make decisions about how to record the charges.
>>>>> > 
>>>>> > There is also the case where an instance is still no longer running but
>>>>> > nova thinks it is (or the reverse), so some sort of auditing sweep needs
>>>>> > to be included (I think that's what Dough called the "farmer" but I
>>>>> > don't have my notes in front of me).
>>>> When I wrote [1], one of the things that I never assumed was how agents
>>>> would collect their information. I imagined that the system should allow
>>>> for multiple implementation of agents that would collect the same
>>>> counters, assuming that 2 implementations for the same counter should
>>>> never be running at once.
>>>> 
>>>> That said, I am not sure an event based collection of what nova is
>>>> notifying would satisfy the requirements I have heard from many cloud
>>>> providers:
>>>> - how do we ensure that event are not forged or lost in the current nova
>>>> system?
>>>> - how can I be sure that an instance has not simply crashed and never
>>>> started?
>>>> - how can I collect information which is not captured by nova events?
>>>> 
>>>> Hence the proposal to use a dedicated event queue for billing, allowing
>>>> for agents to collect and eventually validate data from different
>>>> sources, including, but not necessarily limiting, collection from the
>>>> nova events.
>>>> 
>>>> Moreover, as soon as you generalize the problem to other components than
>>>> just Nova (swift, glance, quantum, daas, ...) just using the nova event
>>>> queue is not an option anymore.
>>>> 
>>>> [1] http://wiki.openstack.org/EfficientMetering
>>>> 
>>>> Nick
>>>> 
>>>> 
>>> 
>>>> On Apr 24, 2012, at 6:20 AM, Sandy Walsh wrote:
>>>> 
>>>>> I think we have support for this currently in some fashion, Dragon?
>>>>> 
>>>>> -S
>>>>> 
>>>>> 
>>>>> 
>>>>> On 04/24/2012 12:55 AM, Loic Dachary wrote:
>>>>>> Metering needs to account for the "volume of data sent to external network destinations " ( i.e. n4 in http://wiki.openstack.org/EfficientMetering ) or the disk I/O etc. This kind of resource is billable.
>>>>>> 
>>>>>> The information described at http://wiki.openstack.org/SystemUsageData will be used by metering but other data sources need to be harvested as well.
>>>> --
>>>> 	Monsyne M. Dragon
>>>> 	OpenStack/Nova 
>>>> 	cell 210-441-0965
>>>> 	work x 5014190
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~openstack
>>>> Post to     : openstack at lists.launchpad.net
>>>> Unsubscribe : https://launchpad.net/~openstack
>>>> More help   : https://help.launchpad.net/ListHelp
>>> 
>>> 
>>> -- 
>>> Loïc Dachary         Chief Research Officer
>>> // eNovance labs   http://labs.enovance.com
>>> // ✉ loic at enovance.com  ☎ +33 1 49 70 99 82
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~openstack
>>> Post to     : openstack at lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~openstack
>>> More help   : https://help.launchpad.net/ListHelp
>> 
>> --
>> Monsyne M. Dragon
>> OpenStack/Nova 
>> cell 210-441-0965
>> work x 5014190
>> 
> 
> 
> -- 
> Loïc Dachary         Chief Research Officer
> // eNovance labs   http://labs.enovance.com
> // ✉ loic at enovance.com  ☎ +33 1 49 70 99 82
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
> 
> 
> 
> 
> -- 
> -------------------------------------------
> Luis Alberto Gervaso Martin
> Woorea Solutions, S.L
> CEO & CTO
> mobile: (+34) 627983344
> luis at woorea.es
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20120424/ee9c1728/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3662 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20120424/ee9c1728/attachment.bin>