[openstack-dev] [oslo][oslo.db][keystone] A POC of Keystone over CockroachDB
mbayer at redhat.com
Wed Sep 6 15:39:52 UTC 2017
On Mon, Sep 4, 2017 at 12:06 PM, Ronan-Alexandre Cherrueau
<ronan-alexandre.cherrueau at inria.fr> wrote:
> Hi folks,
> Recently in the Inria's Discovery initiative, we got in touch with
> CockroachLabs guys with an idea: make Keystone supports CockorachDB. So
> we give it a try and you can find a very first result on our GitHub.
> The GitHub project consists of a Vagrant file that spawns a VM with a
> CockroachDB database and then installs Keystone with Devstack using
> CockroachDB as backend.
> CockroachDB claims to support the PostgreSQL protocol. It also provides
> support for SQLAlchemy that mostly inherits from the PostgreSQL one. So,
> making Keystone working with CockroachDB should be easy peasy right?
> Almost! The rest of this email describes what we have done, to make it
> sqlalchemy-migrate doesn't work with CockroachDB
> Keystone uses a database migration tool named sqlalchemy-migrate.
> This tool is called during the deployment of Keystone to migrate the
> database from its initial versions to its actual version.
> Unfortunately, migration failed with the following (partial)
> | DEBUG migrate.versioning.repository [-] Config:
> OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'),
> ('repository_id', 'keystone'), ('version_table', 'migrate_version'),
> ('required_dbs', ''), ('use_timestamp_numbering', 'False')]))])
> __init__ /usr/local/lib/python2.7/dist-packages/migrate/versioning/repository.py:83
> | INFO migrate.versioning.api [-] 66 -> 67...
> | CRITICAL keystone [-] KeyError: 'cockroachdb'
> | ...
> | TRACE keystone visitorcallable = get_engine_visitor(engine, visitor_name)
> | TRACE keystone File
> line 47, in get_engine_visitor
> | TRACE keystone return get_dialect_visitor(engine.dialect, name)
> | TRACE keystone File
> line 62, in get_dialect_visitor
> | TRACE keystone migrate_dialect_cls = DIALECTS[sa_dialect_name]
> | TRACE keystone KeyError: 'cockroachdb'
> As we understand it, sqlalchemy-migrate contains dedicated SQL backend
> codes and there is no such code for CockroachDB. As a workaround, we
> have performed a second OS deployment with PostgreSQL as backend and
> made a dump of the database after migrations. We then bypassed
> migrations in our first deployment by setting up CockroachDB with the
> dumped script. Note that we had to edit the pgdump script a little bit
> because some features are missing in CockroachDB.
> The workaround we have to go with represents an obstacle to test
> CockroachDB with other OpenStack core services. We would be grateful
> if some of you would help us with adding the support of CockroachDB in
Please don't pursue sqlalchemy-migrate, it is an ancient and
unmaintained project. Keystone should be migrated to Alembic
migrations and this would be a much more useful way to spend the
efforts. Alembic is fully extensible as well and I am highly
motivated to make it work as needed.
> oslo.db contains backend specific code for error handling
> The `oslo.db' library contains one file with backend-specific codes
> (`oslo_db/sqlalchemy/exc_filters.py'). This file aims at
> abstracting database exceptions that differ but target the same class
> of error (because each backend produces a specific exception with a
> specific error message). Intercepted exceptions are wrapped into a
> common OS exception to abstract backend errors in the rest of
> `oslo.db'. CockroachDB doesn't produce same errors than PostgreSQL, so
> we have to update this class. Note that our POC is not exhaustive
> since we only added error messages we saw during Tempest/Rally tests.
> You can look at the differences between OpenStack/oslo.db/stable/pike
> and our fork on GitHub. We only add two lines!
> CockroachDB manages isolation without lock
> CockroachDB lets you write transactions that respect ACID properties.
> Regarding the "I" (Isolation), CockroachDB offers snapshot and
> serializable isolation but doesn't rely on locks to do that. So every
> time there is concurrent editing transactions that end in a conflict,
> then you have to retry the transactions. Fortunately, `oslo.db'
> already offers a decorator to do that automatically. But, based on
> tests we run with Tempest/Rally, we figured out that some Keystone's
> SQL requests needed the decorator.
> You can look at the differences between OpenStack/keystone/stable/pike
> and our fork on GitHub.
> What's next?
> You can drop yourself on the VM as a stack user and run Rally tests
> (see README). Tempest is disabled because we have an issue with its
> installation. Note that modifications we made to make it works are
> minimal. This is promising for the adoption of CockroachDB by other
> core services. Nonetheless:
> - Fixing the problem with sqlalchemy-migrate will be helpful to ease
> the deployment process. If you are willing to help, we will be
> really grateful.
> - We have to look through the performance prism: Keystone over
> CockroachDB vs. Keystone over Galera. This part may involve more
> modifications of `oslo.db'. One thing we have in mind is the
> management of retrying transactions since CockroachDB's
> documentation suggests an approach that doesn't match with actual
> `oslo.db' implementation.
>  https://beyondtheclouds.github.io/
>  https://github.com/BeyondTheClouds/openstack-cockroachdb-dev/
>  https://github.com/openstack/sqlalchemy-migrate/tree/master/
>  https://github.com/openstack/oslo.db/blob/stable/pike/oslo_db/sqlalchemy/exc_filters.py
>  https://github.com/openstack/oslo.db/compare/openstack:stable/pike...BeyondTheClouds:cockroachdb/pike
>  https://github.com/openstack/oslo.db/blob/stable/pike/oslo_db/api.py#L84
>  https://github.com/openstack/keystone/compare/stable/pike...BeyondTheClouds:cockroachdb/pike
>  https://bugs.launchpad.net/devstack/+bug/1714981
>  https://www.cockroachlabs.com/docs/stable/transactions.html#transaction-retries
> Ronan-A. Cherrueau
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
More information about the OpenStack-dev