[Openstack] [ceilometer] Potential New Use Cases

Dan Dyer dan.dyer00 at gmail.com
Wed Nov 7 14:34:39 UTC 2012


Yes, I see a general need to be able to represent meta data the 
identifies associations between monitored systems and between usage 
stored in the datastore. There could be a variety of ways that we need 
to relate event data, so the mechanism should be relatively generic. In 
addition to your use case, we would want to be able to map instances to 
other tenants, group VM's together to represent some kind of shared 
identity or behavior, map instances to some kind of special service 
type. I think there are big advantages to keeping the collection and 
storage of this data separate:
1. It does not require Nova to be aware of the details of the VM internals.
2.It allows for deferral of processing until later in the rating 
process, which helps on scalability
3. makes it easier to extract and report on this data for other use 
cases besides the base billing
4. it more extensible in the sense that you can add arbitrary metadata 
without affecting the core usage data generation.

We use this information as part of our rating process to determine the 
correct charges to apply, so we will need to be able to query for it.

Dan

On 11/5/2012 5:04 AM, Doug Hellmann wrote:
>
>
> On Fri, Nov 2, 2012 at 4:42 PM, Dan Dyer <dan.dyer00 at gmail.com 
> <mailto:dan.dyer00 at gmail.com>> wrote:
>
>     Yes, I am assuming the service controller provides a different
>     stream of data from the lower level VM events. So the question is
>     how to represent and store this additional meta data in
>     ceilometer. Note that there doesn't necessarily need to be a
>     linkage/grouping between the resources since the association is
>     what is actually contained in the metadata that is provided by the
>     service controller.
>
>     As a summary
>     Nova provides its normal events for usage
>     Service controller provides a mapping of nova instances to service
>     type and actual end user
>
>
> So the problem isn't necessarily that you want to measure something 
> different, but that the "ownership" in the existing data is not 
> "correct" from the perspective of the billing system.
>
> We have a similar issue at DreamHost. Our existing user database has 
> account ids that need to be mapped to tenant ids from keystone. Rather 
> than putting that information in keystone, or ceilometer, we decided 
> to store it in our system and have the DreamHost billing system drive 
> the ceilometer API. Does it make sense to do something similar here?
>
> If we definitely want ceilometer to hold the metadata, then I could 
> also see adding an API to let an outside system add metadata to a 
> resource. That would let the PaaS code, which knows about each VM, 
> store extra data that would be returned with the VM metadata when a 
> caller visits /resources/<resourceid>.
>
> Would you expect to be able to query using the metadata? For example, 
> "provide the total instance hours for all instances with paas_tag=foo"?
>
> Doug
>
>
>
>     Dan
>
>
>     On 11/1/2012 11:25 AM, Doug Hellmann wrote:
>>
>>
>>     On Thu, Nov 1, 2012 at 10:21 AM, Dan Dyer <dan.dyer00 at gmail.com
>>     <mailto:dan.dyer00 at gmail.com>> wrote:
>>
>>         In some cases, the service controller is actually running
>>         inside a VM. It would not have access to the internals of the
>>         VM's. It maintains its metadata separately from the Nova
>>         infrastructure.
>>
>>
>>     It doesn't need internal access to the VM, but something has to
>>     share the metadata with ceilometer (or "join" it to the data
>>     ceilometer has) at some point. If it would be too difficult to
>>     get the data into the events, then it could be done by the app
>>     that uses the ceilometer API to query for usage. For example, the
>>     app that loads data from ceilometer to your real billing system
>>     could be driven by data saved by the service controller in
>>     whatever database it uses.
>>
>>     Doug
>>
>>
>>
>>         DD
>>
>>
>>         On 10/25/2012 2:25 AM, Nick Barcet wrote:
>>>         Let's imagine that the service that launch instances can tag the
>>>         instance with:
>>>         a) a common service identifier (constant)
>>>         b) a uuid unique for each "Unit" of the service
>>>         such as <constant>:<uuid>
>>>
>>>         If that tag is passed onto the events which ceilometer stores in its
>>>         entirety as meta, I do not see what the difficulty would be for the
>>>         rating engine to be able to reconcile the information to handle your 2
>>>         use cases.  Am I missing something?
>>>
>>>         Nick
>>>
>>>         On 10/25/2012 12:03 AM, Dan Dyer wrote:
>>>>         I don't think its just a matter of adding more meters or events for a
>>>>         couple of reasons:
>>>>         1. In many cases the metadata I am referring to comes from a different
>>>>         source than the base usage data. Nova is still emitting its normal
>>>>         events, but we get the service/user mapping from a different source. I
>>>>         would not characterize this data as usage metrics but more data about
>>>>         the system relationships.
>>>>         2. in the multiple VM case, we need to have the relationships specified
>>>>         so that we can ignore the proper VM's. There has also been talk of
>>>>         hybrid billing models that charge for some part of the VM usage as well
>>>>         as other metrics. Once again we need a way to characterize the
>>>>         relationships so that processing can associate and filter correctly.
>>>>
>>>>         Dan
>>>>
>>>>         On 10/24/2012 3:35 PM, Julien Danjou wrote:
>>>>>         On Wed, Oct 24 2012, Dan Dyer wrote:
>>>>>
>>>>>>         Use Case 1
>>>>>>         Service Owned Instances
>>>>>>         There are a set of use cases where a service is acting on behalf of a
>>>>>>         user,
>>>>>>         the service is the owner of the VM but billing needs to be attributed
>>>>>>         to the
>>>>>>         end user of the system.This scenario drives two requirements:
>>>>>>         1. Pricing is similar to base VM's but with a premium. So the type of
>>>>>>         service for a VM needs to be identifiable so that the appropriate
>>>>>>         pricing
>>>>>>         can be applied.
>>>>>>         2. The actual end user of the VM needs to be identified so usage can be
>>>>>>         properly attributed
>>>>>         I think that for this, you just need to add more meters on top of the
>>>>>         existing one with your own user and project id information.
>>>>>
>>>>>>         As an example, in some of our PAAS use cases, there is a service
>>>>>>         controller
>>>>>>         running on top of the base VM that maintains the control and and
>>>>>>         manages the
>>>>>>         customer experience. The idea is to expose the service and not have the
>>>>>>         customer have to (or even be able to) manipulate the virtual machine
>>>>>>         directly. So in this case, from a Nova perspective, the PAAS service
>>>>>>         owns
>>>>>>         the VM and it's tenantID is what is reported back in events. The way we
>>>>>>         resolve this is to query the service controller for meta data about that
>>>>>>         instances they own. This is stored off in a separate "table" and used to
>>>>>>         determine the real user at aggregation time.
>>>>>         This is probably where you should emit the meters you need.
>>>>>
>>>>>>         Use Case 2
>>>>>>         Multple Instances combine to make a billable "product/service"
>>>>>>         In this use case, a service might consist of several VM's, but the
>>>>>>         actual
>>>>>>         number does not directly drive the billing.  An example of this might
>>>>>>         be a
>>>>>>         redundant service that has a primary and two backup VM's that make up a
>>>>>>         deployment. The customer is charged for the service, not the fact
>>>>>>         that there
>>>>>>         are 3 VM's running. Once again, we need meta data that is able to
>>>>>>         describe
>>>>>>         this relationship so that when the billing records are processed, this
>>>>>>         relationship can be identified and billed properly.
>>>>>         Kind of the same here, if you don't want to really bill the vm, just
>>>>>         don't meter them (or ignore the meters) and emit your own meter via your
>>>>>         PaaS platform to bill your customer.
>>>>>
>>>>>         Or is there a limitation I miss?
>>>>>
>>>>         _______________________________________________
>>>>         Mailing list:https://launchpad.net/~openstack  <https://launchpad.net/%7Eopenstack>
>>>>         Post to     :openstack at lists.launchpad.net  <mailto:openstack at lists.launchpad.net>
>>>>         Unsubscribe :https://launchpad.net/~openstack  <https://launchpad.net/%7Eopenstack>
>>>>         More help   :https://help.launchpad.net/ListHelp
>>>
>>>
>>>         _______________________________________________
>>>         Mailing list:https://launchpad.net/~openstack  <https://launchpad.net/%7Eopenstack>
>>>         Post to     :openstack at lists.launchpad.net  <mailto:openstack at lists.launchpad.net>
>>>         Unsubscribe :https://launchpad.net/~openstack  <https://launchpad.net/%7Eopenstack>
>>>         More help   :https://help.launchpad.net/ListHelp
>>
>>
>>         _______________________________________________
>>         Mailing list: https://launchpad.net/~openstack
>>         <https://launchpad.net/%7Eopenstack>
>>         Post to     : openstack at lists.launchpad.net
>>         <mailto:openstack at lists.launchpad.net>
>>         Unsubscribe : https://launchpad.net/~openstack
>>         <https://launchpad.net/%7Eopenstack>
>>         More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20121107/5e90da3e/attachment.html>


More information about the Openstack mailing list