Open Stack

Sorry for the delay in getting back on this... comments inline.

On 05/18/2017 06:13 PM, Adrian Turjak wrote:
> Hello fellow OpenStackers,
> 
> For the last while I've been looking at options for multi-region
> multi-master Keystone, as well as multi-master for other services I've
> been developing and one thing that always came up was there aren't many
> truly good options for a true multi-master backend.

Not sure whether you've looked into Galera? We had a geo-distributed 
12-site Galera cluster servicing our Keystone assignment/identity 
information WAN-replicated. Worked a charm for us at AT&T. Much easier 
to administer than master-slave replication topologies and the 
performance (yes, even over WAN links) of the ws-rep replication was 
excellent. And yes, I'm aware Galera doesn't have complete snapshot 
isolation support, but for Keystone's workloads (heavy, heavy read, very 
little write) it is indeed ideal.

(BTW, the cluster setup and node-join operations for CockroachDB and 
Galera are virtually identical...)

 > Recently I've been
> looking at Cockroachdb and while I haven't had the chance to do any
> testing I'm curious if anyone else has looked into it. It sounds like
> the perfect solution, and if it can be proved to be stable enough it
> could solve a lot of problems.
> 
> So, specifically in the realm of Keystone, since we are using sqlalchemy
> we already have Postgresql support, and since Cockroachdb does talk
> Postgres it shouldn't be too hard to back Keystone with it.

OK, now I understand why you didn't consider Galera :) Sounds like 
you're pinned to PostgreSQL for your RDBMS needs...

For the record, CockroachDB doesn't "support PostgreSQL". It supports 
the binary pgsql client/server protocol and, oddly, a view of the 
internal system information in PostgreSQL's pg_catalog schema (though 
also publishes the standard information_schema schema).

The actual *SQL* that CockroachDB uses is definitely not PostgreSQL's 
variant of SQL. CockroachDB's version of SQL is actually pretty close to 
MySQL's version of SQL in a number of ways:

  * EXPLAIN
  * SHOW (TABLES, COLUMNS, CREATE TABLE, DATABASES, etc)
  * RENAME (TABLE, DATABASE, COLUMN, etc)

In other ways, CockroachDB's version of SQL is more like PostgreSQL's 
including:

  * UPSERT (MySQL uses the INSERT ... ON DUPLICATE KEY UPDATE construct)

 > At that
> stage you have a Keystone DB that could be multi-region, multi-master,
> consistent, and mostly impervious to disaster. Is that not the holy
> grail for a service like Keystone? Combine that with fernet tokens and
> suddenly Keystone becomes a service you can't really kill, and can
> mostly forget about.
> 
> I'm welcome to being called mad, but I am curious if anyone has looked
> at this. I'm likely to do some tests at some stage regarding this,
> because I'm hoping this is the solution I've been hoping to find for
> quite a long time.
> 
> Further reading:
> https://www.cockroachlabs.com/
> https://github.com/cockroachdb/cockroach
> https://www.cockroachlabs.com/docs/build-a-python-app-with-cockroachdb-sqlalchemy.html

Another link for folks to read:

https://jepsen.io/analyses/cockroachdb-beta-20160829

I think it's worth investigating and thoroughly testing CockroachDB. 
But, it's pretty new, frankly, and I wouldn't bet a production system on 
it. Also, please do follow up on the performance of CockroachDB, which 
as aphyr notes in the jepsen link above, was far, far below other RDBMS 
that have been tested.

Best,
-jay