Open Stack

On 5/14/15 5:44 AM, John Garbutt wrote:
> On 14 May 2015 at 07:18, Manickam, Kanagaraj <kanagaraj.manickam at hp.com> wrote:
>> Hi Nova team,
>>
>> This mail is regarding an help required on the migration from
>> sqlalchemy migration tool  to alembic tool.
>>
>> Heat is currently using sqlalchemy-migration tool and In liberty release,
>> we are investigating on how to bring the alembic into heat. We found that
>> nova has already tried the same (https://review.openstack.org/#/c/15196/ )
>> almost 2 years back and in Kilo release, nova is still using sqlalchemy
>> migration tool.
>> (https://github.com/openstack/nova/tree/master/nova/db/sqlalchemy/migrate_repo/versions)
>>
>> So we are assuming that, in nova, you might have faced blockers to bring in
>> alembic.
>> And we would like to seek your  recommendation/suggestions based
>> on your experience on this.  This will help us to take proper direction
>> on using alembic in heat. Could you kindly share it.
> We are currently looking at going this direction (its not merged yet though):
> http://specs.openstack.org/openstack/nova-specs/specs/kilo/approved/online-schema-changes.html
>
> If we get that working, we wouldn't need to write or generate any
> migrations, we end up doing a "diff" between the desired state and the
> current database state, and split that into expand and contract
> phases.
>
> To support that approach, we are no longer doing any data migrations
> in our migration scripts, they are happening in a different way,
> "online":
> http://specs.openstack.org/openstack/nova-specs/specs/kilo/approved/flavor-from-sysmeta-to-blob.html#other-deployer-impact
>
> There are more details on how this is all planned to fit together here:
> http://docs.openstack.org/developer/nova/devref/upgrade.html
>
> Anyways, this is why we have paused the alembic vs sqlalchemy debate
> for the moment.

The "online schema changes" patch has been abandoned.    I regret that I 
was not able to review the full nature of this spec in time to note some 
concerns I have, namely that Alembic does not plan on ever acheiving 
100% "automation" of migration generation; such a thing is not possible 
and would require a vast amount of development resources in any case to 
constantly keep up with the ever changing features and behaviors of all 
target databases.   The online migration spec, AFAICT, does not offer 
any place for manual migrations to be added, and I think this will be a 
major problem. The decisions made by "autogenerate" I have always stated 
should always be manually reviewed and corrected, so I'd be very nervous 
about a system that uses autogenerate on the fly and sends those changes 
directly to a production database without any review.

FWIW in Nova I am planning to look into doing a simple one-shot of the 
Nova migration scripts in nova/db/sqlalchemy/migrate_repo/versions to 
Alembic.   The only large migration script in this repo is 
216_havana.py, which consists of all Table defs that do not need any 
changes.  There are 39 remaining scripts, and all of these are extremely 
short and simple, and just a direct manual change to Alembic style is 
most expedient for these.

The more intellectual part of the exercise is to replace the use of 
sqlalchemy-migrate APIs in Nova's installation scripts and test 
harnesses to use Alembic as well.   At that point Nova will be on a 
straight Alembic install that will at first behave similarly to the 
Migrate one.

Assuming I can get that done in the next few months, the next step would 
be that the migration streams can be broken into branches, e.g. juno, 
kilo, liberty, etc. so that we can easily add new migration files that 
are backportable in place to a target backport.   Additionally, the 
issue of end-user migrations that are divergent from what is part of 
Nova itself should also be achieved using additional branches; if a 
customer wants to make their own changes to a database, such as adding 
extra indexes, this can be done in their own user-specific branch that 
will integrate with the primary migration stream.    That would 
eliminate the need for a system that is tasked with diffing live 
database schemas and then executing decisions directly without any 
chance to review or version-control the changes that are being made.











>
> Does that help?
>
> Thanks,
> John
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev