Open Stack

Hi Brian,

On Fri, Feb 18, 2011 at 01:21:52PM -0500, Brian Lamar wrote:
> +1 vote for Python/C modules for needed 'performance'.
> 
> I might be the only one thinking that Python is up to the task, but I wanted to throw that out there.

The main reason I don't see Python as a contender is the GIL. Because
of it, leveraging all cores means running multi-process, and while this
is suitable for some applications (Swift, Nova, ...), it's not suitable
for a queue. Services like Swift and Nova are good use cases because
there isn't a need for cross-communication for two accepted sockets
(which may be handled in separate processes) for a given server. With
a queue, there is. One socket may be pushing jobs, and another socket
may be pulling those same jobs. If we're in the same process this is
very efficient, but when these clients are in different processes, you
need to rely on IPC mechanisms. IPC is never going to be as efficient
as in-process message passing (with the exception of Solaris doors, but
that doesn't help us here :), and the extra overhead from IPC means we
won't be gaining much even running it multi-process in the first place.

So, in short, because of the cross-communication nature of the queue
clients, we need something with efficient single-process and multi-core
abilities, and Python doesn't give us that.

> C++ and I have a history and I'd rather everything be written in C (yeah, I'm one of those people). For truly complex tasks C + Python work very well together and can produce some amazing results.
> 
> That being said, C++ is a solid choice.

Hehe, understood. I was primarily a C server developer for 10+
years, and that version of me would not be happy with my suggestion of
C++. Over the past couple years though I've found when you're trying to
build an open source community around a project, the simpler and more
bullet-proof you can make your plugin APIs, the better. Giving someone
a C header file where they need to think about memory allocation,
setting callbacks, and so on is much more error prone than extending a
base plugin class and implementing pure virtual methods. For a small
project C would be a good fit, but for larger, modular projects with
hopes of engaging many external developers, C++ makes a good case.

> Languages aside, it looks like you've got a decent start going, Eric, with Scale Stack. I do worry about reinventing the wheel a bit, although I'm not clear on the goals of your project so I could be wrong.

The main goals of Scale Stack are emphasizing code/module reuse,
easy to write extensions (all the dependency tracking and autotool
bootstrapping is taken care of), and providing a solid multi-core
foundation. The event frameworks/libs out there are single-core
(libevent, libev, node.js, eventlet, twisted, ...), where Scale Stack
has a module that handles the multi-core issues for you. When you add
event handlers they are distributed around to all cores configured
for the event plugin. It also has all the networking related modules
ready to go.

Erlang already does this too (multi-core event loop and much more),
which is why I mentioned it as a good option.

> Could this service just be an aggregation of some other proven technologies? The first thing that came to mind when this service was suggested was using a proven scalable key/value store to provide a queue-like interface and then all we'd have to do is provide the interface on top of it.

Yes, and with whatever language we choose, this will be the case. We
need to write the "kernel" and provide plugin interfaces to write
things like backing stores for any number of existing storage
solutions, public protocol modules that may leverage existing
libraries, auth modules that use LDAP, and so on.

-Eric


> -----Original Message-----
> From: "Eric Day" <eday at oddments.org>
> Sent: Thursday, February 17, 2011 5:21pm
> To: openstack at lists.launchpad.net
> Subject: [Openstack] Queue Service, next steps
> 
> Thanks to everyone who gave great feedback on the first queue service
> thread. I've updated the wiki page to include the suggestions.
> 
> http://wiki.openstack.org/QueueService
> 
> With a decent vision of what we want to build, the next step is
> figuring out how. In a previous thread it was suggested that the
> preferred languages for OpenStack projects are Python, C, and
> C++. Since there is an emphasis on speed and efficiency for the
> queue service, I suggest we use C++. I expect this service to be
> CPU bound and would benefit being able to leverage multiple cores
> efficiently (within the same process), so I don't think Python
> is a good fit. I think C++ is a better fit than C due to the need
> for modular interfaces. While this can obviously be done in C, C++
> APIs are more concise and much less error prone. The OO style will
> also make it easier for Python developers who also want to learn and
> assist with C++ projects.
> 
> Erlang is not on the preferred lists, but I would also put it out
> there as an option. While it may be a great fit for a project like
> this, I worry it won't attract the developer resources since Erlang
> isn't really a first-class language yet.
> 
> If we decide to take the C++ path, I propose using a modular
> application framework I've been working on over the past year (mostly
> in my spare time). It provides a simple module programming interface
> with dependency tracking (kind of like Linux kernel modules). It
> already provides a multi-threaded event module (currently based on
> libevent, but this is pluggable) with simple networking abstractions
> built on top of it. We should be able to dive in an start writing the
> HTTP protocol module and queue processing modules. You can check out
> the current project at:
> 
> https://launchpad.net/scalestack
> http://scalestack.org/
> 
> The intention of using a framework like this is so we can easily reuse
> the other modules (auth, HTTP, logging, ...) for other OpenStack
> services in the future. Much like we use Eventlet, WSGI, etc. (and
> eventually openstack-common) for Python, we could prefer using the
> modules in Scale Stack for lower level projects.
> 
> Thoughts?
> -Eric
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>