Open Stack

Hi, Openstackers

I would like to propose a new initiative to implement AWS DynamoDB API for
OpenStack.

I believe this is a right moment for OpenStack community to start MagnetoDB
discussion and design around such service as more data processing and
management services, such as Savanna and Trove, are appearing in OpenStack,
creating a sufficient basis for higher level data APIs like DynamoDB API.
With Trove you can provision and manage different databases including NoSQL
databases support, which will appear soon. It minimizes the maintenance
process and turns database deployment and management to API calls.

DynamoDB [1] <http://aws.amazon.com/dynamodb/> is well-known key-value
NoSQL storage provided by Amazon with HTTP interface. It uses JSON as its
transfer data model. The data is stored as key-value, where value is list
of attributes. The attribute’s value can be String, Number, Binary or sets
of those types.  Read consistency can be strong or eventual per request.
The secondary indexes are also supported.

Simplicity, reliability, and pleasant documentation have made it actively
used in a lot of applications, but hosting them on OpenStack is impossible
due to the absence of such storage. DynamoDB is already widely used and has
its own niche [2] <http://aws.amazon.com/dynamodb/testimonials/> despite
being closed and proprietary.

Let me share the idea to implement a similar service which imitates
DynamoDB API [3]<http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/API.html>,
called MagnetoDB.

MagnetoDB will fully implement DynamoDB API thus enabling the possibility
for cloud users to migrate their applications that use DynamoDB from AWS to
OpenStack.

For backend storage provisioning and management, OpenStack DBaaS Trove will
be used.

The backend database should be pluggable to provide flexibility in terms
choosing the solution which best matched the existing or planned OpenStack
installation technology stack. MySQL is one of the obvious options, being
the de-facto standard for use with OpenStack and being supported by Trove
right away.

However, users of DynamoDB expect high performance, scalability, and
availability. Those can be achieved more easily  by using one of the NoSQL
solutions in MagnetoDB.

Apache Cassandra [4] <http://cassandra.apache.org/>looks very suitable for
that case, due to its tunable consistency, easy scalability, key-value data
model, ring topology and other features that give us predictable high
performance and fault tolerance. The Cassandra data model perfectly fits
MagnetoDB needs.

Moreover, support of NoSQL databases is in Trove roadmap
[5]<https://blueprints.launchpad.net/trove/+spec/cassandra-db-support>and
the fist version of Cassandra provisioning is currently in progress
[6] <https://review.openstack.org/#/c/51884/>.

AWS SDK [7]<http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/UsingAWSSDK.html>can
be used to query data. Provided by Amazon, it hides all protocol stuff
giving the high level API.  In Python world you can use the rather common
Boto [8] <https://github.com/boto/boto>
[9]<http://boto.readthedocs.org/en/latest/dynamodb2_tut.html>library.

An  initial draft of the more official proposal is available here:
https://wiki.openstack.org/wiki/MagnetoDB

Any contribution, comments and pieces of advice are very welcome

Also, I plan to make the lightning talk on Tuesday, November 5 at Expo
Breakout Room 1.

See you on summit.



[1] http://aws.amazon.com/dynamodb

[2] http://aws.amazon.com/dynamodb/testimonials/

[3] http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/API.html

[4] http://cassandra.apache.org/

[5] https://blueprints.launchpad.net/trove/+spec/cassandra-db-support

[6] https://review.openstack.org/#/c/51884/

[7]
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/UsingAWSSDK.html

[4] https://github.com/boto/boto

[5] http://boto.readthedocs.org/en/latest/dynamodb2_tut.html

With best regards,
Ilya Sviridov

<http://www.mirantis.ru/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131030/aa11154a/attachment.html>