Open Stack

Hello openstackers,

We are working on MagnetoDB project and trying our best to follow 
OpenStack standards.


So, MagnetoDB is aimed to be high performance scalable OpenStack based 
WSGI application which provide interface to high available distributed 
reliable key-value storage. We investigated best practices and separated 
the next points:

 1.

    to avoid problems with GIL our application should be executed in
    single thread mode with non-blocking IO (using greenlets or another
    python specific approaches to rich this)

 2.

    to make MagnetoDB scalable it is necessary to make MagnetoDB
    stateless. It allows us run a lot of independent MagnetoDB processes
    and switch all requests flow between them:

     1.

        at single node to load all CPU's cores

     2.

        at the different nodes for horizontal scalability

 3.

    use Cassandra as most reliable and mature distributed key-value storage

 4.

    use datastax python-driver as most modern cassandra python client
    which supports newest CQL3 and Cassandra native binary protocol
    features set


So, considering this points The next technologies was chosen:

 1.

    gevent as one of the fastest non-blocking single-thread WSGI server.
    It is based on greenlet library and supports monkey patching of
    standard threading library. It is necessary because of datastax
    python driver uses threading library and it's backlog has task to
    add gevent backlog. (We patched python-driver ourselves to enable
    this feature as temporary solution and waiting for new python-driver
    releases). It makes gevent more interesting to use than other
    analogs (like eventlet for example)

 2.

    gunicorn as WSGI server which is able to run a few worker processes
    and master process for workers managing and routing request between
    them. Also it has integration with gevent and can run gevent based
    workers. We also analyzed analogues, such as uWSGI. It looks like
    more faster but unfortunately we didn't manage to work uWSGI in
    multi process mode with MagnetoDB application.


Also I want to add that currently oslo wsgi framework is used for 
organizing request routing. I know that current OpenStack trend is to 
migrate WSGI services to Pecan wsgi framework. Maybe is it reasonable 
for MagnetoDB too.


We would like to hear your opinions about the libraries and approaches 
we have chosen and would appreciate you help and support in order to 
find the best balance between performance, developer friendness  and 
OpenStack standards.

-- 
Best regards,
Dmitriy Ukhlov
Mirantis Inc.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140318/e95f38e5/attachment.html>