Open Stack

On Tue, Mar 18, 2014 at 2:51 PM, Dmitriy Ukhlov <dukhlov at mirantis.com>wrote:

>  Hello openstackers,
>
>  We are working on MagnetoDB project and trying our best to follow
> OpenStack standards.
>
>  So, MagnetoDB is aimed to be high performance scalable OpenStack based
> WSGI application which provide interface to high available distributed
> reliable key-value storage. We investigated best practices and separated
> the next points:
>
>    1.
>
>    to avoid problems with GIL our application should be executed in
>    single thread mode with non-blocking IO (using greenlets or another python
>    specific approaches to rich this)
>     2.
>
>    to make MagnetoDB scalable it is necessary to make MagnetoDB
>    stateless. It allows us run a lot of independent MagnetoDB processes and
>    switch all requests flow between them:
>     1.
>
>       at single node to load all CPU's cores
>        2.
>
>       at the different nodes for horizontal scalability
>        3.
>
>    use Cassandra as most reliable and mature distributed key-value storage
>     4.
>
>    use datastax python-driver as most modern cassandra python client
>    which supports newest CQL3 and Cassandra native binary protocol features set
>
>
>  So, considering this points The next technologies was chosen:
>
>    1.
>
>    gevent as one of the fastest non-blocking single-thread WSGI server.
>    It is based on greenlet library and supports monkey patching of standard
>    threading library. It is necessary because of datastax python driver uses
>    threading library and it's backlog has task to add gevent backlog. (We
>    patched python-driver ourselves to enable this feature as temporary
>    solution and waiting for new python-driver releases). It makes gevent more
>    interesting to use than other analogs (like eventlet for example)
>     2.
>
>    gunicorn as WSGI server which is able to run a few worker processes
>    and master process for workers managing and routing request between them.
>    Also it has integration with gevent and can run gevent based workers. We
>    also analyzed analogues, such as uWSGI. It looks like more faster but
>    unfortunately we didn't manage to work uWSGI in multi process mode with
>    MagnetoDB application.
>
>
> Also I want to add that currently oslo wsgi framework is used for
> organizing request routing. I know that current OpenStack trend is to
> migrate WSGI services to Pecan wsgi framework. Maybe is it reasonable for
> MagnetoDB too.
>

The WSGI framework in Oslo is officially deprecated, and should no longer
be used for new projects. Its use in existing projects is being phased out
as projects create backwards-incompatible API changes that make moving to
new tools safe (there's no point in creating issues by rewriting the old
services). You should not use the old WSGI framework if you plan to propose
MagenetoDB for incubation.

Several integrated and incubated projects have already adopted Pecan, so
there are quite a few examples available to get you started. Drop by
#pecanpy on freenode if you have questions.

Doug



>
> We would like to hear your opinions about the libraries and approaches we
> have chosen and would appreciate you help and support in order to find the
> best balance between performance, developer friendness  and OpenStack
> standards.
>
> --
> Best regards,
> Dmitriy Ukhlov
> Mirantis Inc.
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140319/e7046a66/attachment.html>