[openstack-dev] [magnetodb] Using gevent in MagnetoDB. OpenStack standards and approaches

Ryan Petrello ryan.petrello at dreamhost.com
Wed Mar 19 13:49:34 UTC 2014


Gunicorn + gevent + pecan play nicely together, and they’re a combination I’ve used to good success in the past.  Pecan even comes with some helpers for integrating with gunicorn:

$ gunicorn_pecan pecan_config.py -k gevent -w4

Ryan Petrello
Senior Developer, DreamHost
ryan.petrello at dreamhost.com

On Mar 18, 2014, at 2:51 PM, Dmitriy Ukhlov <dukhlov at mirantis.com> wrote:

> Hello openstackers,
> We are working on MagnetoDB project and trying our best to follow OpenStack standards.
> So, MagnetoDB is aimed to be high performance scalable OpenStack based WSGI application which provide interface to high available distributed reliable key-value storage. We investigated best practices and separated the next points:
> 	• to avoid problems with GIL our application should be executed in single thread mode with non-blocking IO (using greenlets or another python specific approaches to rich this)
> 	• to make MagnetoDB scalable it is necessary to make MagnetoDB stateless. It allows us run a lot of independent MagnetoDB processes and switch all requests flow between them:
> 		• at single node to load all CPU’s cores
> 		• at the different nodes for horizontal scalability
> 	• use Cassandra as most reliable and mature distributed key-value storage
> 	• use datastax python-driver as most modern cassandra python client which supports newest CQL3 and Cassandra native binary protocol features set
> So, considering this points The next technologies was chosen:
> 	• gevent as one of the fastest non-blocking single-thread WSGI server. It is based on greenlet library and supports monkey patching of standard threading library. It is necessary because of datastax python driver uses threading library and it’s backlog has task to add gevent backlog. (We patched python-driver ourselves to enable this feature as temporary solution and waiting for new python-driver releases). It makes gevent more interesting to use than other analogs (like eventlet for example)
> 	• gunicorn as WSGI server which is able to run a few worker processes and master process for workers managing and routing request between them. Also it has integration with gevent               and can run gevent based workers. We also analyzed analogues, such as uWSGI. It looks like more faster but unfortunately we didn’t manage to work uWSGI in multi process mode with MagnetoDB application.
> Also I want to add that currently oslo wsgi framework is used for organizing request routing. I know that current OpenStack trend is to migrate WSGI services to Pecan wsgi framework. Maybe is it reasonable for MagnetoDB too.
> We would like to hear your opinions about the libraries and approaches we have chosen and would appreciate you help and support in order to find the best balance between performance, developer friendness  and OpenStack standards.
> -- 
> Best regards,
> Dmitriy Ukhlov
> Mirantis Inc.
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

More information about the OpenStack-dev mailing list