Open Stack

On Tue, 2013-01-15 at 10:52 +0000, Daniel P. Berrange wrote:
> This email is in relation to the following bug
> 
>   https://bugs.launchpad.net/nova/+bug/1099761
> 
> If a guest administrator shuts down their VM from inside the guest
> OS using 'shutdown -h now' (or equivalent), it will take Nova upto
> 10 minutes to notice that the VM has stopped running
> 
> The Nova<->hypervisor state synchronization is done in a periodic
> task (nova.compute.manager._sync_power_states) which only runs
> every 10 minutes. It used to be every 60 seconds until this bug
> was addressed:
> 
>   https://bugs.launchpad.net/nova/+bug/928910
> 
> I put most of the blame for that bug's perforance issues in the use
> of the wrong libvirt APIs, but none the less I don't really want to
> put the periodic task frequency back down to 60 seconds.
> 
> IMHO the key problem here is the design, not the implementation. In
> particular the use of polling by the manager to detect the state
> changes is inherantly inefficient. Libvirt provides an async event
> notification mechanism allowing applications to have a callback
> triggered whenever a guest changes lifecycle state. Other hypervisor
> APIs like ESX have similar notification mechanisms. Making use of
> this would allow state changes to be detected pretty much immediately.
> 
> The question is how to structure the processing of events. I don't
> think that the hypervisor drivers should be directly processing events.
> Instead I believe they need to pass along the event notifications to
> the manager.py class. So my current thought is to introduce a new API
> to nova.virt.api
> 
>   register_event_notifier(self, callback)
> 
> and have nova.compute.manager provide a callback impl to receive the
> events. Before I start coding on this, I want some kind of confirmation
> that this is an acceptable direction to go in, since there is no current
> callback based interactions between nova.compute.manager & nova.virt.api

That all sounds good to me, but can you go into more details about how
the event dispatching would work?

We don't have a mainloop to integrate this with, so presumably you're
talking about spawning a greenthread which would poll for events (using
virEventRunDefaultImpl()?) and then invoke the callbacks? Does this
greenthread need its own libvirt connection?

Does this new thread introduce any new concurrency issues? I guess not
since processing each RPC message and running periodic timers all happen
in different threads, so this wouldn't be much different.

Cheers,
Mark.