Open Stack

On 07/16/2013 01:58 PM, Boris Pavlovic wrote:

> There is no "magic" and we have only 2 ways to end up with this problems
> and bugs that could be caused by "manually" migrations merging and tons
> of bugs in sqlalchemy-migrate.
>
> 1) step by step (openstack method)
>    There are special tests "test_migrations" that runs migrations on
> real data against all backends. So we should:
>
>    a) improve this tests to checks all behaviors // there is a lot of
> hidden bugs
>    b) replace migration (only one small migration) to alembic
>    c) check that in all backends we made the same changes in schema
>    d) Merge all old migrations in one using alembic (automatically).
>    So it could be done in safe way.
>
> 2.a) huge 2 steps
>    1. Merge all migrations in one huge manually (drop all tests in test
> migrations)
>        e.g. In Nova was patch https://review.openstack.org/#/c/35748/
>        I don't believe that there are no mistakes in this migration, and
> nobody is able to check it. // because of tons of hidden bugs in old
> migrations and sqla-migrate.
>    2. Replace this migration in Alembic
>         I don't believe that there will be way to check that there is no
> bugs
>
> 2.b) suicide mode (1 big step)
>    Merge and switch in one step=)

We have compacted migrations before, and there's a test document for how 
to verify that the big migration has exactly the same output as the 
series of small migrations.  See 
https://wiki.openstack.org/wiki/Database_migration_testing  Dan Prince 
is the expert on this.

I think the right process is:

1. Wait until the very beginning of Icehouse cycle.  (But not after we 
have new migrations for Icehouse.)

2. Compact all migrations into 2xx_havana.py (for SQLAlchemy-migrate)

3. Test that it's perfect via above test plan plus whatever enhancements 
we think of.

4. Manually convert 2xx_havana.py (for SQLAlchemy-migrate) into Alembic, 
and verify that it's still perfect.

5. Deprecate the SQLAlchemy-migrate version and announce that new 
migrations should be in Alembic.

#4 is hard work but not impossible.  I have some old code that does 90% 
of the work, so we only have to do the other 90%.

-- 
David Ripton   Red Hat   dripton at redhat.com