Open Stack

On 01/15/2013 07:04 AM, Mark McLoughlin wrote:
> On Tue, 2013-01-15 at 10:52 +0000, Daniel P. Berrange wrote:
>> This email is in relation to the following bug
>>
>>   https://bugs.launchpad.net/nova/+bug/1099761
>>
>> If a guest administrator shuts down their VM from inside the guest
>> OS using 'shutdown -h now' (or equivalent), it will take Nova upto
>> 10 minutes to notice that the VM has stopped running
>>
>> The Nova<->hypervisor state synchronization is done in a periodic
>> task (nova.compute.manager._sync_power_states) which only runs
>> every 10 minutes. It used to be every 60 seconds until this bug
>> was addressed:
>>
>>   https://bugs.launchpad.net/nova/+bug/928910
>>
>> I put most of the blame for that bug's perforance issues in the use
>> of the wrong libvirt APIs, but none the less I don't really want to
>> put the periodic task frequency back down to 60 seconds.
>>
>> IMHO the key problem here is the design, not the implementation. In
>> particular the use of polling by the manager to detect the state
>> changes is inherantly inefficient. Libvirt provides an async event
>> notification mechanism allowing applications to have a callback
>> triggered whenever a guest changes lifecycle state. Other hypervisor
>> APIs like ESX have similar notification mechanisms. Making use of
>> this would allow state changes to be detected pretty much immediately.
>>
>> The question is how to structure the processing of events. I don't
>> think that the hypervisor drivers should be directly processing events.
>> Instead I believe they need to pass along the event notifications to
>> the manager.py class. So my current thought is to introduce a new API
>> to nova.virt.api
>>
>>   register_event_notifier(self, callback)
>>
>> and have nova.compute.manager provide a callback impl to receive the
>> events. Before I start coding on this, I want some kind of confirmation
>> that this is an acceptable direction to go in, since there is no current
>> callback based interactions between nova.compute.manager & nova.virt.api
> 
> That all sounds good to me, but can you go into more details about how
> the event dispatching would work?
> 
> We don't have a mainloop to integrate this with, so presumably you're
> talking about spawning a greenthread which would poll for events (using
> virEventRunDefaultImpl()?) and then invoke the callbacks? Does this
> greenthread need its own libvirt connection?
> 
> Does this new thread introduce any new concurrency issues? I guess not
> since processing each RPC message and running periodic timers all happen
> in different threads, so this wouldn't be much different.

Assuming details are worked out and all sound good, this all sounds
great to me so far.

Thanks,

-- 
Russell Bryant