Open Stack

On 08/27/2013 11:22 AM, Sandy Walsh wrote:
> On 08/27/2013 05:32 AM, Boris Pavlovic wrote:
>> Jay,
>>
>> I should probably share to you about our work around DB.
>>
>> Migrations should be run only in production and only for production
>> backends (e.g. psql and mysql)
>> In tests we should use Schemas created by Models
>> (BASE.metadata.create_all())
>>
>> We are not able to use in this approach in moment  because we don't have
>> any mechanism to check that MODELS and SCHEMAS are EQUAL.
>> And actually MODELS and SCHEMAS are DIFFERENT.
>>
>> E.g. in Celiometer we have BP that syncs models and migration
>> https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-db-sync-models-with-migrations
>> (in other projects we are doing the same)
>>
>> And also we are working around (oslo) generic tests that checks that
>> models and migrations are equal:
>> https://review.openstack.org/#/c/42307/
>>
>>
>> So in our roadmap (in this case is):
>> 1) Soft switch to alembic (with code that allows to have sqla-migrate
>> and alembic migration in the same time)
>> 2) Sync Models and Migrations (fix DB schemas also)
>> 3) Add from oslo generic test that checks all this stuff
>> 4) Use BASE.create_all() for Schema creation instead of migrations.
>>
>>
>> But in OpenStack is not so simple to implement such huge changes, so it
>> take some time=)
>
> Hmm, so I'm not sure how to proceed here?
>
> Should we be using alembic or the older migrations right now?
>
> Are there any examples of what you're proposing here?

Yeah, basically I am just confused in the case of Ceilometer, 
specifically. There are sqlalchemy-migrate migrations [1] and then there 
are Alembic migrations [2] all in the same project.

I have no idea how they are supposed to work together, but it seems like 
the SQLalchemy migrations are supposed to be run before the Alembic 
migrations [3]. However, I don't think the Alembic migrations have even 
been tested, frankly. [4]

What I believe is an appropriate course of action would be to do a 
single patch (hopefully BEFORE Havana is released) that would:

a) Remove all SQLAlchemy-migrate migrations
b) Create an initial Alembic migration script that would set the 
database state to the model state that should have existed at the "end" 
of the existing SQLAlchemy migrations (currently, migration 012)
c) Have the initial Alembic migration remove the sqlalchemy-migrate 
versioning table and history, if it exists.
d) Remove the existing code in the base storage test case that calls 
conn.upgrade() in the setUp() method (!)
e) Have the base storage test case for the sqlalchemy driver simply call 
sqlalchemy.MetaData.create_all() instead of running any migrations
f) Have EVERY Alembic migration tested individually with a database 
schema that has actual data in it (like is done in Glance with the 
sqlalchemy-migrate migrations)

That said, Alembic has no support for common ALTER TABLE operations with 
SQLite [5], and so I recommend having the migration tests in f) above 
only tested on production database engines (MySQL, PostgreSQL, etc).

Thoughts?
-jay

[1] 
https://github.com/openstack/ceilometer/tree/master/ceilometer/storage/sqlalchemy/migrate_repo/versions
[2] 
https://github.com/openstack/ceilometer/tree/master/ceilometer/storage/sqlalchemy/alembic/versions
[3] 
https://github.com/openstack/ceilometer/blob/master/ceilometer/storage/sqlalchemy/migration.py#L68
[4] https://bugs.launchpad.net/ceilometer/+bug/1217156
[5] 
https://bitbucket.org/zzzeek/alembic/issue/21/column-renames-not-supported-on-sqlite