Open Stack

On Tue, Nov 20, 2012 at 11:59:24AM +0800, Huang Zhiteng wrote:
> Agree that the root cause is GIL.
> I don't like the way we are struggling with pony threads (whatever you
> may call eventlet) to stuff a whole bunch of work into _single_
> process, which means we can only utilize one processor while we may
> have several dozens of them.  It's era of many-core already!  That
> doesn't make sense and eventually doesn't scale (think of a situation
> when compute node have > 100 cores, which may to real in just a few
> years).
> 
> Another service or fork new process for periodic task, they may solve
> the problem for a while, but the ultimate solution is true
> parallelism.  I'd like to take one more step forward: how about
> working out a new framework for periodic tasks that can put each task
> into its own *true* thread/process(worker?)?

I can't help but agree with this. We're already jumping through
silly hoops to try and stop C API calls into libraries like libvirt
blocking, which is something that 'just works' when using real OS
threads. Not to mention adding various hacks in various other code
to "yeild" control to other threads. All problems of our own making
and I can't see it getting any easier.

Yes, you could just run 8 instances of nova-compute per host if
you have 8 CPUs in it, but this approach to scalability is just
wasteful of memory. IMHO if we want to have a long term scalable
architecture for series, we need to make use of real OS threads
in python.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|