[openstack-dev] RFC: Synchronizing hypervisor <-> nova state with event notifications

Russell Bryant rbryant at redhat.com
Tue Jan 15 13:00:51 UTC 2013

On 01/15/2013 07:04 AM, Mark McLoughlin wrote:
> On Tue, 2013-01-15 at 10:52 +0000, Daniel P. Berrange wrote:
>> This email is in relation to the following bug
>>   https://bugs.launchpad.net/nova/+bug/1099761
>> If a guest administrator shuts down their VM from inside the guest
>> OS using 'shutdown -h now' (or equivalent), it will take Nova upto
>> 10 minutes to notice that the VM has stopped running
>> The Nova<->hypervisor state synchronization is done in a periodic
>> task (nova.compute.manager._sync_power_states) which only runs
>> every 10 minutes. It used to be every 60 seconds until this bug
>> was addressed:
>>   https://bugs.launchpad.net/nova/+bug/928910
>> I put most of the blame for that bug's perforance issues in the use
>> of the wrong libvirt APIs, but none the less I don't really want to
>> put the periodic task frequency back down to 60 seconds.
>> IMHO the key problem here is the design, not the implementation. In
>> particular the use of polling by the manager to detect the state
>> changes is inherantly inefficient. Libvirt provides an async event
>> notification mechanism allowing applications to have a callback
>> triggered whenever a guest changes lifecycle state. Other hypervisor
>> APIs like ESX have similar notification mechanisms. Making use of
>> this would allow state changes to be detected pretty much immediately.
>> The question is how to structure the processing of events. I don't
>> think that the hypervisor drivers should be directly processing events.
>> Instead I believe they need to pass along the event notifications to
>> the manager.py class. So my current thought is to introduce a new API
>> to nova.virt.api
>>   register_event_notifier(self, callback)
>> and have nova.compute.manager provide a callback impl to receive the
>> events. Before I start coding on this, I want some kind of confirmation
>> that this is an acceptable direction to go in, since there is no current
>> callback based interactions between nova.compute.manager & nova.virt.api
> That all sounds good to me, but can you go into more details about how
> the event dispatching would work?
> We don't have a mainloop to integrate this with, so presumably you're
> talking about spawning a greenthread which would poll for events (using
> virEventRunDefaultImpl()?) and then invoke the callbacks? Does this
> greenthread need its own libvirt connection?
> Does this new thread introduce any new concurrency issues? I guess not
> since processing each RPC message and running periodic timers all happen
> in different threads, so this wouldn't be much different.

Assuming details are worked out and all sound good, this all sounds
great to me so far.


Russell Bryant

More information about the OpenStack-dev mailing list