Open Stack

On 6/17/2015 10:52 AM, Matt Riedemann wrote:
> Without getting into the details from the etherpad [1], a few of us in
> IRC today were talking about how the ceilometer compute-agent polls
> libvirt directly for guest VM statistics and how ceilometer should
> really be getting this information from nova via notifications sent from
> a periodic task in the nova compute manager.
>
> Nova already has the get_instance_diagnostics virt driver API which is
> nice in that it has structured versioned instance diagnostic information
> regardless of virt driver (unlike the v2 os-server-diagnostics API which
> is a free-form bag of goodies depending on which virt driver is used,
> which makes it mostly untestable and not portable).  The problem is the
> get_instance_diagnostics virt driver API is per-instance, so it's not
> efficient in the case that you want bulk instance data for a given
> compute host.
>
> So the idea is to add a new virt driver API to get the bulk data and
> emit that via a structured versioned payload similar to
> get_instance_diagnostics but for all instances.
>
> Eventually the goal is for nova to send what ceilometer is collecting
> today [2] and then ceilometer can just consume that notification rather
> than doing the direct hypervisor polling it has today.
>
> Anyway, this is the high level idea, the details/notes are in the
> etherpad along with next steps.
>
> Feel free to chime in now with reasons why this is crazy and will never
> work and we shouldn't waste our time on it.
>
> [1] https://etherpad.openstack.org/p/nova-hypervisor-bulk-stats-notify
> [2]
> http://docs.openstack.org/admin-guide-cloud/content/section_telemetry-compute-meters.html
>
>

Waking up from a rare nap opportunity on a Saturday, this is what was 
bothering me:

The proposal in the etherpad assumes that we are just getting bulk 
host/domain/guest VM stats from the hypervisor and sending those in a 
notification, but how do we go about filtering those out to only 
instances that were booted through Nova?

Jason pointed out the ceilometer code gets all of the non-error state 
instances from nova first [1] and then for each of those it does the 
domain lookup from libvirt, filtering out any that are in SHUTOFF state [2].

When talking about the new virt driver API for bulk stats, danpb said to 
use virConnectGetAllDomainStats with libvirt [3] but I'm not aware of 
that being able to filter out instances that weren't created by nova.  I 
don't think we want a notification from nova about the hypervisor stats 
to include things that were created outside nova, like directly through 
virsh or vCenter.

For at least libvirt, if virConnectGetAllDomainStats returns the domain 
metadata then we can filter those since there is nova-specific metadata 
in the domains created through nova [4] but I'm not sure that's true 
about the other virt types in nova (I think the vCenter driver tags VMs 
somehow as being created by OpenStack/Nova, but not sure about 
xen/hyper-v/ironic).

I guess adding support for something like bulk guest VM stats in a nova 
virt driver would have a pre-requisite of being able to uniquely 
identify guest VMs from the hypervisor as being created by nova.

Otherwise we'd just basically be moving the same thing that ceilometer 
is doing today into nova, as a per-instance loop and then using the 
os-server-diagnostics virt driver calls from the compute manager (at 
least saving us a compute API call from ceilometer every 10 minutes).

Thoughts?

[1] 
https://github.com/openstack/ceilometer/blob/master/ceilometer/compute/discovery.py#L35
[2] 
https://github.com/openstack/ceilometer/blob/master/ceilometer/compute/virt/libvirt/inspector.py#L111
[3] 
http://libvirt.org/html/libvirt-libvirt-domain.html#virConnectGetAllDomainStats
[4] http://paste.openstack.org/show/308305/

-- 

Thanks,

Matt Riedemann