Open Stack

Here's a previous effort at alembic:
https://review.openstack.org/#/c/150057/

-- Dims

On Wed, Sep 6, 2017 at 11:39 AM, Michael Bayer <mbayer at redhat.com> wrote:
> On Mon, Sep 4, 2017 at 12:06 PM, Ronan-Alexandre Cherrueau
> <ronan-alexandre.cherrueau at inria.fr> wrote:
>> Hi folks,
>>
>> Recently in the Inria's Discovery initiative[1], we got in touch with
>> CockroachLabs guys with an idea: make Keystone supports CockorachDB. So
>> we give it a try and you can find a very first result on our GitHub[2].
>> The GitHub project consists of a Vagrant file that spawns a VM with a
>> CockroachDB database and then installs Keystone with Devstack using
>> CockroachDB as backend.
>>
>> CockroachDB claims to support the PostgreSQL protocol. It also provides
>> support for SQLAlchemy that mostly inherits from the PostgreSQL one. So,
>> making Keystone working with CockroachDB should be easy peasy right?
>> Almost! The rest of this email describes what we have done, to make it
>> works.
>>
>>
>> sqlalchemy-migrate doesn't work with CockroachDB
>> ================================================
>>
>> Keystone uses a database migration tool named sqlalchemy-migrate[3].
>> This tool is called during the deployment of Keystone to migrate the
>> database from its initial versions to its actual version.
>> Unfortunately, migration failed with the following (partial)
>> stacktrace:
>>
>> ,----
>> | DEBUG migrate.versioning.repository [-] Config:
>> OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'),
>> ('repository_id', 'keystone'), ('version_table', 'migrate_version'),
>> ('required_dbs', '[]'), ('use_timestamp_numbering', 'False')]))])
>> __init__ /usr/local/lib/python2.7/dist-packages/migrate/versioning/repository.py:83
>> | INFO migrate.versioning.api [-] 66 -> 67...
>> | CRITICAL keystone [-] KeyError: 'cockroachdb'
>> | ...
>> | TRACE keystone     visitorcallable = get_engine_visitor(engine, visitor_name)
>> | TRACE keystone   File
>> "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py",
>> line 47, in get_engine_visitor
>> | TRACE keystone     return get_dialect_visitor(engine.dialect, name)
>> | TRACE keystone   File
>> "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py",
>> line 62, in get_dialect_visitor
>> | TRACE keystone     migrate_dialect_cls = DIALECTS[sa_dialect_name]
>> | TRACE keystone KeyError: 'cockroachdb'
>> `----
>>
>> As we understand it, sqlalchemy-migrate contains dedicated SQL backend
>> codes and there is no such code for CockroachDB. As a workaround, we
>> have performed a second OS deployment with PostgreSQL as backend and
>> made a dump of the database after migrations. We then bypassed
>> migrations in our first deployment by setting up CockroachDB with the
>> dumped script. Note that we had to edit the pgdump script a little bit
>> because some features are missing in CockroachDB.
>>
>> The workaround we have to go with represents an obstacle to test
>> CockroachDB with other OpenStack core services. We would be grateful
>> if some of you would help us with adding the support of CockroachDB in
>> sqlalchemy-migrate.
>
> Please don't pursue sqlalchemy-migrate, it is an ancient and
> unmaintained project.   Keystone should be migrated to Alembic
> migrations and this would be a much more useful way to spend the
> efforts.  Alembic is fully extensible as well and I am highly
> motivated to make it work as needed.
>
>
>>
>>
>> oslo.db contains backend specific code for error handling
>> =========================================================
>>
>> The `oslo.db' library contains one file with backend-specific codes
>> (`oslo_db/sqlalchemy/exc_filters.py'[4]). This file aims at
>> abstracting database exceptions that differ but target the same class
>> of error (because each backend produces a specific exception with a
>> specific error message). Intercepted exceptions are wrapped into a
>> common OS exception to abstract backend errors in the rest of
>> `oslo.db'. CockroachDB doesn't produce same errors than PostgreSQL, so
>> we have to update this class. Note that our POC is not exhaustive
>> since we only added error messages we saw during Tempest/Rally tests.
>>
>> You can look at the differences between OpenStack/oslo.db/stable/pike
>> and our fork on GitHub[5]. We only add two lines!
>>
>>
>> CockroachDB manages isolation without lock
>> ==========================================
>>
>> CockroachDB lets you write transactions that respect ACID properties.
>> Regarding the "I" (Isolation), CockroachDB offers snapshot and
>> serializable isolation but doesn't rely on locks to do that. So every
>> time there is concurrent editing transactions that end in a conflict,
>> then you have to retry the transactions. Fortunately, `oslo.db'
>> already offers a decorator[6] to do that automatically. But, based on
>> tests we run with Tempest/Rally, we figured out that some Keystone's
>> SQL requests needed the decorator.
>>
>> You can look at the differences between OpenStack/keystone/stable/pike
>> and our fork on GitHub[7].
>>
>>
>> What's next?
>> ============
>>
>> You can drop yourself on the VM as a stack user and run Rally tests
>> (see README). Tempest is disabled because we have an issue with its
>> installation[8]. Note that modifications we made to make it works are
>> minimal. This is promising for the adoption of CockroachDB by other
>> core services. Nonetheless:
>>
>> - Fixing the problem with sqlalchemy-migrate will be helpful to ease
>>   the deployment process. If you are willing to help, we will be
>>   really grateful.
>>
>> - We have to look through the performance prism: Keystone over
>>   CockroachDB vs. Keystone over Galera. This part may involve more
>>   modifications of `oslo.db'. One thing we have in mind is the
>>   management of retrying transactions since CockroachDB's
>>   documentation suggests an approach[9] that doesn't match with actual
>>   `oslo.db' implementation.
>>
>> Best,
>>
>> [1] https://beyondtheclouds.github.io/
>> [2] https://github.com/BeyondTheClouds/openstack-cockroachdb-dev/
>> [3] https://github.com/openstack/sqlalchemy-migrate/tree/master/
>> [4] https://github.com/openstack/oslo.db/blob/stable/pike/oslo_db/sqlalchemy/exc_filters.py
>> [5] https://github.com/openstack/oslo.db/compare/openstack:stable/pike...BeyondTheClouds:cockroachdb/pike
>> [6] https://github.com/openstack/oslo.db/blob/stable/pike/oslo_db/api.py#L84
>> [7] https://github.com/openstack/keystone/compare/stable/pike...BeyondTheClouds:cockroachdb/pike
>> [8] https://bugs.launchpad.net/devstack/+bug/1714981
>> [9] https://www.cockroachlabs.com/docs/stable/transactions.html#transaction-retries
>>
>> --
>> Ronan-A. Cherrueau
>> https://rcherrueau.github.io
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims