Open Stack

Hello,

Please see my comments inline.

Best regards,
Alexei Kornienko

On 07/16/2013 09:16 PM, David Ripton wrote:
> On 07/16/2013 01:58 PM, Boris Pavlovic wrote:
>
>> There is no "magic" and we have only 2 ways to end up with this problems
>> and bugs that could be caused by "manually" migrations merging and tons
>> of bugs in sqlalchemy-migrate.
>>
>> 1) step by step (openstack method)
>>    There are special tests "test_migrations" that runs migrations on
>> real data against all backends. So we should:
>>
>>    a) improve this tests to checks all behaviors // there is a lot of
>> hidden bugs
>>    b) replace migration (only one small migration) to alembic
>>    c) check that in all backends we made the same changes in schema
>>    d) Merge all old migrations in one using alembic (automatically).
>>    So it could be done in safe way.
>>
>> 2.a) huge 2 steps
>>    1. Merge all migrations in one huge manually (drop all tests in test
>> migrations)
>>        e.g. In Nova was patch https://review.openstack.org/#/c/35748/
>>        I don't believe that there are no mistakes in this migration, and
>> nobody is able to check it. // because of tons of hidden bugs in old
>> migrations and sqla-migrate.
>>    2. Replace this migration in Alembic
>>         I don't believe that there will be way to check that there is no
>> bugs
>>
>> 2.b) suicide mode (1 big step)
>>    Merge and switch in one step=)
>
> We have compacted migrations before, and there's a test document for 
> how to verify that the big migration has exactly the same output as 
> the series of small migrations.  See 
> https://wiki.openstack.org/wiki/Database_migration_testing  Dan Prince 
> is the expert on this.
>
> I think the right process is:
>
> 1. Wait until the very beginning of Icehouse cycle.  (But not after we 
> have new migrations for Icehouse.)
>
> 2. Compact all migrations into 2xx_havana.py (for SQLAlchemy-migrate)
As Boris mentioned: "We are able only to convert only grizzly migrations 
not havana (because our customers should be able to switch from grizzly 
to havana)"
so in the beginning of Icehouse we'll have:
2xx_grizzly.py (sqla-migrate)
2xx_havana.py (sqla-migrate)
...
2xx_havana.py (sqla-migrate)
___________________
Icehouse migrations...

So my question is: when we should start using alembic for new migrations?
We won't be able to deprecate sqla-migrate and switch to alembic in one 
step since we'll have some migrations kept for backwards compatibility.
I think that to accomplish your plan we should start using alembic for 
new migrations starting from Icehouse.
If you are ok with it I can change ceilometer migrations implementation 
to work accordingly and we'll be able to use it as a reference in other 
projects.

>
> 3. Test that it's perfect via above test plan plus whatever 
> enhancements we think of.
>
> 4. Manually convert 2xx_havana.py (for SQLAlchemy-migrate) into 
> Alembic, and verify that it's still perfect.
>
> 5. Deprecate the SQLAlchemy-migrate version and announce that new 
> migrations should be in Alembic.
>
> #4 is hard work but not impossible.  I have some old code that does 
> 90% of the work, so we only have to do the other 90%.
>