Open Stack

Hi Li,

How are you going to make this separation transparent? I mean,
generally, in a function code, you can't know in advance if the
transaction will be read-only or it will contain an
INSERT/UPDATE/DELETE statement. On the other hand, as a developer, you
could analyze the DB queries that can be possibly issued by this
function and mark the function somehow, so that oslo.db would know for
which database connection the transaction should be created, but this
is essentially what slave_connection option is for and how it works
now.

Secondly, as you said, the key thing here is to separate reads and
writes. In order to make reads fast/reduce the load on your 'writable'
database, you'd move reads to asynchronous replicas. But you can't do
this transparently either, as there is a lot of places in our code, in
which we assume we are using the latest state of data, while
asynchronous replicas might actually be a little bit out of date. So,
in case of slave_connection, we use it only when it's ok for the code
to work with outdated rows, i.e. *explicitly* modify the existing
functions to work with slave_connection.

Thanks,
Roman

On Fri, Aug 8, 2014 at 7:03 AM, Li Ma <skywalker.nick at gmail.com> wrote:
> Getting a massive amount of information from data storage to be displayed is
> where most of the activity happens in OpenStack. The two activities of reading
> data and writing (creating, updating and deleting) data are fundamentally
> different.
>
> The optimization for these two opposite database activities can be done by
> physically separating the databases that service these two different
> activities. All the writes go to database servers, which then replicates the
> written data to the database server(s) dedicated to servicing the reads.
>
> Currently, AFAIK, many OpenStack deployment in production try to take
> advantage of MySQL (includes Percona or MariaDB) multi-master Galera cluster.
> It is possible to design and implement a read/write separation schema
> for such a DB cluster.
>
> Actually, OpenStack has a method for read scalability via defining
> master_connection and slave_connection in configuration, but this method
> lacks of flexibility due to deciding master or slave in the logical
> context(code). It's not transparent for application developer.
> As a result, it is not widely used in all the OpenStack projects.
>
> So, I'd like to propose a transparent read/write separation method
> for oslo.db that every project may happily takes advantage of it
> without any code modification.
>
> Moreover, I'd like to put it in the mailing list in advance to
> make sure it is acceptable for oslo.db.
>
> I'd appreciate any comments.
>
> br.
> Li Ma
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev