Open Stack

On Tue, May 20, 2014, Collins, Sean <Sean_Collins2 at cable.comcast.com> wrote:
> I've been looking at two reviews that Ann Kamyshnikova has proposed
> 
> https://review.openstack.org/#/c/82073/
> 
> https://review.openstack.org/#/c/80518/
> 
> I think the changes are fundamentally a Good Thing™  - they appear to
> reduce the differences between the database models and their
> corresponding migrations – as well as fixing differences in the
> generated DDL between Postgres and MySQL.
> 
> The only thing I'm concerned about, is how to prevent these
> inconsistencies from sneaking into the codebase in the future. The
> one review that fixes ForeignKey constraints that are missing a name
> argument which ends up failing to create indexes in Postgres – I can
> see myself repeating that mistake, it's very subtle.
> 
> Should we have some sort of HACKING for database models and
> migrations? Are these problems subtle enough that they warrant
> changes to SQLAlchemy/Alembic?

On the Nova side of things, there has been similar concerns.

There is a nova-spec that is proposing adding a unit test to check the
schema versus the model:

https://review.openstack.org/#/c/85325/

This should work, but I think the underlying problem is DRY based. We
should not need to declare a schema in a model and then a set of
imperative tasks to get to that point. All too often they get
unsynchronized.

I informally proposed a different solution, moving schema migrations to
a declarative model. I wrote a proof of concept to show how something
like this would work:

https://github.com/jerdfelt/muscle

We already have a model written (but need some fixes to make it accurate
wrt to existing migrations), we should be able to emit ALTER TABLE
statements based on the existing schema to bring it into line with the
model.

This also has the added benefit of allowing certain schema migrations to
be done online, while services are still running. This can significantly
reduce downtime during deploys (a big concern for large deployments of
Nova).

There are some corner cases that do cause problems (renaming columns,
changing column types, etc). Those can either remain as traditional
migrations and/or discouraged.

Data migrations would still remain with sqlalchemy-migrate/alembic, but
there have some proposals about solving that problem too.

JE