[openstack-dev] RFC: Synchronizing hypervisor <-> nova state with event notifications

Mark McLoughlin markmc at redhat.com
Fri Jan 18 07:08:30 UTC 2013


On Tue, 2013-01-15 at 16:41 +0000, Daniel P. Berrange wrote:
> On Tue, Jan 15, 2013 at 04:18:55PM +0000, Mark McLoughlin wrote:
> > On Tue, 2013-01-15 at 13:58 +0000, Daniel P. Berrange wrote:
> > > On Tue, Jan 15, 2013 at 01:45:53PM +0000, Mark McLoughlin wrote:
> > > > On Tue, 2013-01-15 at 13:24 +0000, Daniel P. Berrange wrote:
> > > > > On Tue, Jan 15, 2013 at 12:04:31PM +0000, Mark McLoughlin wrote:
> > > > > > On Tue, 2013-01-15 at 10:52 +0000, Daniel P. Berrange wrote:
> > > > > > > The question is how to structure the processing of events. I don't
> > > > > > > think that the hypervisor drivers should be directly processing events.
> > > > > > > Instead I believe they need to pass along the event notifications to
> > > > > > > the manager.py class. So my current thought is to introduce a new API
> > > > > > > to nova.virt.api
> > > > > > > 
> > > > > > >   register_event_notifier(self, callback)
> > > > > > > 
> > > > > > > and have nova.compute.manager provide a callback impl to receive the
> > > > > > > events. Before I start coding on this, I want some kind of confirmation
> > > > > > > that this is an acceptable direction to go in, since there is no current
> > > > > > > callback based interactions between nova.compute.manager & nova.virt.api
> > > > > > 
> > > > > > That all sounds good to me, but can you go into more details about how
> > > > > > the event dispatching would work?
> > > > > > 
> > > > > > We don't have a mainloop to integrate this with, so presumably you're
> > > > > > talking about spawning a greenthread which would poll for events (using
> > > > > > virEventRunDefaultImpl()?) and then invoke the callbacks? Does this
> > > > > > greenthread need its own libvirt connection?
> > > > > > 
> > > > > > Does this new thread introduce any new concurrency issues? I guess not
> > > > > > since processing each RPC message and running periodic timers all happen
> > > > > > in different threads, so this wouldn't be much different.
> > > > > 
> > > > > Yes, thanks to eventlet awfulness, we'd need to have a greenthread running
> > > > > the libvirt event loop.  The event notification callbacks from libvirt
> > > > > would thus obviously execute in the greenthread. What I'm not sure about
> > > > > is how to switch control back to the main eventlet thread before we call
> > > > > out into the manager.py's callback.
> > > > >
> > > > > I imagine a queue of events in the nova.virt.libvirt.driver object which
> > > > > is fed back the libvirt greenthread callbacks. Something in the main
> > > > > eventlet thread would then have to process the queue to dispatch to the
> > > > > manager. With a regular mainloop the way you'd do this is to schedule a
> > > > > timer to fire after zero seconds. I'm not familiar enough with eventlet
> > > > > yet to know how you'd do this.
> > > > 
> > > > There isn't really a "main eventlet thread" - the main greenthread is
> > > > just sitting there waiting for the other greenthreads to finish.
> > > > 
> > > > The greenthread for consuming RPC messages spawns off a greenthread for
> > > > each message and this is the normal entry point into manager.py.
> > > > 
> > > > That's why I'm thinking just invoking the callback directly from your
> > > > greenthread would work.
> > > > 
> > > > (I'm explicitly saying greenthread over and over because it's so easy to
> > > > forget these aren't native threads :)
> > > 
> > > Of course when I wrote 'greenthread' in my description above, I of course
> > > mean native thread. You can't invoke libvirt API calls directly from
> > > greenthreads because native code blocks the whole interpretor unless you
> > > use native threads.
> > >
> > > So the libvirt callback will be running in a native thread and needs to
> > > pass control back to a greenthread for processing. I guess I can have a
> > > dedicated greenthread that processes a queue of events from the native
> > > thread.
> > 
> > Hmm, not sure I buy that.
> > 
> > If this was a single-threaded app with a mainloop, I think I'd be happy
> > to add libvirt's "check for events" timer to the main thread. Do you
> > really see the timer blocking for that long?
> 
> To receive events from libvirt you need to provide an event loop impl
> to libvirt, in the form of running this code:
> 
>   libvirt.virEventRegisterDefaultImpl()
>   while True:
>      libvirt.virEventRunDefaultImpl()
> 
> to allow libvirt to process incoming events from libvirtd. While those
> API calls above are invoked from a greenthread, since we use a native
> threadpool for all libvirt APIs, this means they're going to actually
> switch to a native to run the C code. This implies that the callback
> from libvirt is going to be invoked in the context of a native thread.
> We thus need to get out of native thread context & into a greenthread
> context before we can pass the event back up to the nova.compute.manager
> class' callback.

FWIW, that makes sense to me - I'd missed the fact that we use a native
threadpool for libvirt API calls.

Cheers,
Mark.




More information about the OpenStack-dev mailing list