Open Stack

From: Eoghan Glynn [eglynn at redhat.com]
Sent: Friday, November 23, 2012 8:17 AM

>> I've outlined a variety of proposals for solving this problem using
>> the existing infrastructure. There's not a large departure needed
>> from where we are today.
>>
>> In a nutshell, for lifecycle/metering purposes:
>> 1. The existing notifiers are sufficient (they don't have to use
>> AMQP)
>
>Right, they don't.
>
>> 2. The notification event should be structured with optional
>> components and versioned accordingly.
>
>OK.
>
>> 3. The ceilometer worker pulls in much of the nova common rpc
>> baggage, which is unnecessary (notifications are not rpc calls).
>> Look at the StackTach worker for an example of something lean and
>> mean.
>
>That's a fair point, but orthogonal to the question of relying
>entirely on notifications.

Notifications are just a place in the code where hooks activated. What you do with them is an exercise for the reader. It's a jumping-off-point. A way to keep the underlying code from becoming too cluttered. 

Honestly I see several things happening here:
a. the underlying notification structure changing as in #2 above.
b. a new amqp notification driver that takes this new structure and reformats it to be backward compatible with existing billing solutions/deployments.
c. a new amqp notification driver that can write to a completely different amqp system ... away from the production amqp system. 

This will allow to have backwards compatibility today and a means for growth down the road.

>> 4. The existing usage notifications for bandwidth and state seem fine
>> (I haven't heard objection use-cases). Any exceptions I've heard
>> have been instrumentation or user-space based.
>
>OK, by the notifications for bandwidth, do you mean the info extracted
>via ComputeDriver.get_all_bw_counters()?

Quite likely, I need to dig into that to confirm. Once the TX guys get back from the turkeyfest I'll ask them about the bandwidth usage approach. My understanding is it's getting reported every 10 minutes. The rest of the info is available in a common manner as in:
https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2764
https://github.com/openstack/nova/blob/master/nova/virt/xenapi/driver.py#L556

We should extend this to approach to unify whatever is being done for bw under xen.

>I may have misunderstood what's going on there, but that seemed quite
>unsuitable to me as it currently exists.
>
>Basically the problems are:
>
>1. this is only currently supported for the xenapi driver (libvirt
>   raises NotImplementedError).

minor

>2. the bandwidth stats are cached in the DB then retrieved again before
>   being reported, so the mechanism is quite heavyweight.

Well, b/w accounting is the biggest issue with billing. Getting it right is critically important for SP's. Personally I like the comfort of having it in the db for later retrieval vs having it float in the ether until it gets picked up. Somewhere, someone has to persist this stuff quickly. Should this be done in Nova ... Hmm. I could see both arguments. It is a pretty important thing though. I suspect it lives there because it was done when ceilometer was but a glint-in-the-eye.

What would a lighter weight approach look like?

>3. the trigger for reporting the bandwidth are non-periodic instance
>   lifecycle events like an update/resize/rebuild, or the low-frequency
>  instance auditing (runs at most once an hour).

Let me confirm the "every 10 minutes" thing I heard. Could be a future.

The code is here:
https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2832

>None of these issues are necessarily un-fixable, but I don't think
>the mechanism is suitable for use out of the box.
>
>Of course I'm open to correction if I mis-read the above issues in
>nova, or if you had a different bandwidth notification mechanism
>in mind.
>
>> 5. There is a hook in the virt layer for getting info out. We could
>> easily extend this to include optional drivers from ceilometer for
>> customer specific needs.

>Just to clarify, is that hook the get_all_bw_counters() driver API?

See above.

(and thanks again for the reply Eoghan ... it often feels like this discussion falls on deaf ears)

-S


>Cheers,
>Eoghan