Open Stack

On Mon, Jan 14, 2013 at 09:22:51PM +1100, Michael Still wrote:
> On Mon, Jan 14, 2013 at 9:11 PM, Daniel P. Berrange <berrange at redhat.com> wrote:
> > On Sat, Jan 12, 2013 at 07:28:02PM +1100, Michael Still wrote:
> 
> >> Now that the periodic tasks refactor is done, I've just sent off a
> >> code review which implements a external periodic task runner. Its at
> >> https://review.openstack.org/#/c/19539/
> >
> > This thread has quite a large number of objections to the proposed
> > introduction of this new service, which AFAIK, were never resolved.
> > IMHO it is premature to be considering a merge of the feature until
> > there is broad agreement that it is a good idea.
> 
> So, I re-read the thread, and there's basically one objection --
> eventlet isn't true parallelism and is causing us pain. Correct me if
> I'm wrong.

Yes, that's pretty much assumed to be the root cause, though there may
be higher level issues we're unaware of making life worse than required.

> It seems unlikely to me that we're going to get rid of eventlet in
> grizzly, or even that doing that is a good idea. There are lots of
> assumptions about eventlet in the code base -- for example our locking
> implementation will probably need reworking if we drop eventlet. I'd
> welcome a prototype patch that demonstrated swapping out eventlet if
> you'd like to contribute one.
> 
> So, there are people having problems with scale right now. What are
> these people supposed to do if we can't replace eventlet anytime soon,
> but you're not willing to move some periodic tasks out into another
> process?

Well there are a number of possible options that could be explored.

There may be some things we're doing with eventlet that are sub-optimal
and could be improved - AFAIK no one has looked in detail at what is
causing the scalability problems. eg perhaps there are some specific
pieces of Nova code that could be changed to mitigate the problems.
For example in the libvirt driver we use a threadpool for running the
native API calls to avoid blocking eventlet. This doc

  doc/source/devref/threading.rst

indicates that DB calls used to do the same thing, but apparently don't
anymore. That said, this may be solved with most of the DB calls being
removed due to nodb-compute blueprint. There could be other areas that
would benefit from a threadpool though

If threadpools don't suffice, rather than introducing explicit new user
visible binaries, it may be preferrable to just have the main binary
fork() some dedicated processes for periodic tasks. Alternatively it
may actually be preferrable to fork() 'n' complete nova-compute services,
one for each logical CPU in the host to maximise scaling across the
entire service, not merely periodic tasks.

Or maybe nova-periodic is the right idea after all - I'm just not sure
based on the limit info presented thus far.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|