[openstack-dev] [nova][ceilometer] proposal to send bulk hypervisor stats data in periodic notifications
mriedem at linux.vnet.ibm.com
Sat Jun 20 18:50:53 UTC 2015
On 6/17/2015 10:52 AM, Matt Riedemann wrote:
> Without getting into the details from the etherpad , a few of us in
> IRC today were talking about how the ceilometer compute-agent polls
> libvirt directly for guest VM statistics and how ceilometer should
> really be getting this information from nova via notifications sent from
> a periodic task in the nova compute manager.
> Nova already has the get_instance_diagnostics virt driver API which is
> nice in that it has structured versioned instance diagnostic information
> regardless of virt driver (unlike the v2 os-server-diagnostics API which
> is a free-form bag of goodies depending on which virt driver is used,
> which makes it mostly untestable and not portable). The problem is the
> get_instance_diagnostics virt driver API is per-instance, so it's not
> efficient in the case that you want bulk instance data for a given
> compute host.
> So the idea is to add a new virt driver API to get the bulk data and
> emit that via a structured versioned payload similar to
> get_instance_diagnostics but for all instances.
> Eventually the goal is for nova to send what ceilometer is collecting
> today  and then ceilometer can just consume that notification rather
> than doing the direct hypervisor polling it has today.
> Anyway, this is the high level idea, the details/notes are in the
> etherpad along with next steps.
> Feel free to chime in now with reasons why this is crazy and will never
> work and we shouldn't waste our time on it.
>  https://etherpad.openstack.org/p/nova-hypervisor-bulk-stats-notify
Waking up from a rare nap opportunity on a Saturday, this is what was
The proposal in the etherpad assumes that we are just getting bulk
host/domain/guest VM stats from the hypervisor and sending those in a
notification, but how do we go about filtering those out to only
instances that were booted through Nova?
Jason pointed out the ceilometer code gets all of the non-error state
instances from nova first  and then for each of those it does the
domain lookup from libvirt, filtering out any that are in SHUTOFF state .
When talking about the new virt driver API for bulk stats, danpb said to
use virConnectGetAllDomainStats with libvirt  but I'm not aware of
that being able to filter out instances that weren't created by nova. I
don't think we want a notification from nova about the hypervisor stats
to include things that were created outside nova, like directly through
virsh or vCenter.
For at least libvirt, if virConnectGetAllDomainStats returns the domain
metadata then we can filter those since there is nova-specific metadata
in the domains created through nova  but I'm not sure that's true
about the other virt types in nova (I think the vCenter driver tags VMs
somehow as being created by OpenStack/Nova, but not sure about
I guess adding support for something like bulk guest VM stats in a nova
virt driver would have a pre-requisite of being able to uniquely
identify guest VMs from the hypervisor as being created by nova.
Otherwise we'd just basically be moving the same thing that ceilometer
is doing today into nova, as a per-instance loop and then using the
os-server-diagnostics virt driver calls from the compute manager (at
least saving us a compute API call from ceilometer every 10 minutes).
More information about the OpenStack-dev