[openstack-dev] [DB][Migrations] Switching to using of Alembic

Doug Hellmann doug.hellmann at dreamhost.com
Wed Jul 17 12:48:41 UTC 2013

On Wed, Jul 17, 2013 at 6:38 AM, Alexei Kornienko <
alexei.kornienko at gmail.com> wrote:

> Hello,
> Please see my comments inline.
> Best regards,
> Alexei Kornienko
> On 07/16/2013 09:16 PM, David Ripton wrote:
>> On 07/16/2013 01:58 PM, Boris Pavlovic wrote:
>>  There is no "magic" and we have only 2 ways to end up with this problems
>>> and bugs that could be caused by "manually" migrations merging and tons
>>> of bugs in sqlalchemy-migrate.
>>> 1) step by step (openstack method)
>>>    There are special tests "test_migrations" that runs migrations on
>>> real data against all backends. So we should:
>>>    a) improve this tests to checks all behaviors // there is a lot of
>>> hidden bugs
>>>    b) replace migration (only one small migration) to alembic
>>>    c) check that in all backends we made the same changes in schema
>>>    d) Merge all old migrations in one using alembic (automatically).
>>>    So it could be done in safe way.
>>> 2.a) huge 2 steps
>>>    1. Merge all migrations in one huge manually (drop all tests in test
>>> migrations)
>>>        e.g. In Nova was patch https://review.openstack.org/#**/c/35748/<https://review.openstack.org/#/c/35748/>
>>>        I don't believe that there are no mistakes in this migration, and
>>> nobody is able to check it. // because of tons of hidden bugs in old
>>> migrations and sqla-migrate.
>>>    2. Replace this migration in Alembic
>>>         I don't believe that there will be way to check that there is no
>>> bugs
>>> 2.b) suicide mode (1 big step)
>>>    Merge and switch in one step=)
>> We have compacted migrations before, and there's a test document for how
>> to verify that the big migration has exactly the same output as the series
>> of small migrations.  See https://wiki.openstack.org/**
>> wiki/Database_migration_**testing<https://wiki.openstack.org/wiki/Database_migration_testing> Dan Prince is the expert on this.
>> I think the right process is:
>> 1. Wait until the very beginning of Icehouse cycle.  (But not after we
>> have new migrations for Icehouse.)
>> 2. Compact all migrations into 2xx_havana.py (for SQLAlchemy-migrate)
> As Boris mentioned: "We are able only to convert only grizzly migrations
> not havana (because our customers should be able to switch from grizzly to
> havana)"
> so in the beginning of Icehouse we'll have:
> 2xx_grizzly.py (sqla-migrate)
> 2xx_havana.py (sqla-migrate)
> ...
> 2xx_havana.py (sqla-migrate)
> ___________________
> Icehouse migrations...
> So my question is: when we should start using alembic for new migrations?
> We won't be able to deprecate sqla-migrate and switch to alembic in one
> step since we'll have some migrations kept for backwards compatibility.
> I think that to accomplish your plan we should start using alembic for new
> migrations starting from Icehouse.
> If you are ok with it I can change ceilometer migrations implementation to
> work accordingly and we'll be able to use it as a reference in other
> projects.

As far as I'm concerned for ceilometer, we can start using alembic for
migrations as soon as the dbsync command runs both sets of migrations in
the right sequence. You have a patch in review for that now, so I don't
think we'll need to wait for Icehouse.


>> 3. Test that it's perfect via above test plan plus whatever enhancements
>> we think of.
>> 4. Manually convert 2xx_havana.py (for SQLAlchemy-migrate) into Alembic,
>> and verify that it's still perfect.
>> 5. Deprecate the SQLAlchemy-migrate version and announce that new
>> migrations should be in Alembic.
>> #4 is hard work but not impossible.  I have some old code that does 90%
>> of the work, so we only have to do the other 90%.
> ______________________________**_________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.**org <OpenStack-dev at lists.openstack.org>
> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130717/28de2586/attachment.html>

More information about the OpenStack-dev mailing list