<div dir="ltr"><div>Hi!</div><div><br></div><div>It is nice to see so careful attention for my change requests. In fact they are related the same topic that Johannes mentioned. The idea of this is described there <a href="https://blueprints.launchpad.net/neutron/+spec/db-sync-models-with-migrations">https://blueprints.launchpad.net/neutron/+spec/db-sync-models-with-migrations</a>. </div>
<div><br></div><div>Problems that mentioned in <a href="https://review.openstack.org/#/c/82073/">https://review.openstack.org/#/c/82073/</a> <a href="https://review.openstack.org/#/c/80518/">https://review.openstack.org/#/c/80518/</a> were discovered with the help of this test <a href="https://review.openstack.org/74081">https://review.openstack.org/74081</a> that I adopted for Neutron <a href="https://review.openstack.org/76520">https://review.openstack.org/76520</a>. It helps to find a lot of bugs in Neutron database and models. Now as this change <a href="https://review.openstack.org/74081">https://review.openstack.org/74081</a> is going to move to oslo.db all of this work is frozen.</div>
<div><br></div><div>Regards,</div><div>Ann</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, May 20, 2014 at 9:02 AM, Johannes Erdfelt <span dir="ltr"><<a href="mailto:johannes@erdfelt.com" target="_blank">johannes@erdfelt.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Tue, May 20, 2014, Collins, Sean <<a href="mailto:Sean_Collins2@cable.comcast.com">Sean_Collins2@cable.comcast.com</a>> wrote:<br>
> I've been looking at two reviews that Ann Kamyshnikova has proposed<br>
><br>
> <a href="https://review.openstack.org/#/c/82073/" target="_blank">https://review.openstack.org/#/c/82073/</a><br>
><br>
> <a href="https://review.openstack.org/#/c/80518/" target="_blank">https://review.openstack.org/#/c/80518/</a><br>
><br>
> I think the changes are fundamentally a Good Thing™ - they appear to<br>
> reduce the differences between the database models and their<br>
> corresponding migrations – as well as fixing differences in the<br>
> generated DDL between Postgres and MySQL.<br>
><br>
> The only thing I'm concerned about, is how to prevent these<br>
> inconsistencies from sneaking into the codebase in the future. The<br>
> one review that fixes ForeignKey constraints that are missing a name<br>
> argument which ends up failing to create indexes in Postgres – I can<br>
> see myself repeating that mistake, it's very subtle.<br>
><br>
> Should we have some sort of HACKING for database models and<br>
> migrations? Are these problems subtle enough that they warrant<br>
> changes to SQLAlchemy/Alembic?<br>
<br>
</div></div>On the Nova side of things, there has been similar concerns.<br>
<br>
There is a nova-spec that is proposing adding a unit test to check the<br>
schema versus the model:<br>
<br>
<a href="https://review.openstack.org/#/c/85325/" target="_blank">https://review.openstack.org/#/c/85325/</a><br>
<br>
This should work, but I think the underlying problem is DRY based. We<br>
should not need to declare a schema in a model and then a set of<br>
imperative tasks to get to that point. All too often they get<br>
unsynchronized.<br>
<br>
I informally proposed a different solution, moving schema migrations to<br>
a declarative model. I wrote a proof of concept to show how something<br>
like this would work:<br>
<br>
<a href="https://github.com/jerdfelt/muscle" target="_blank">https://github.com/jerdfelt/muscle</a><br>
<br>
We already have a model written (but need some fixes to make it accurate<br>
wrt to existing migrations), we should be able to emit ALTER TABLE<br>
statements based on the existing schema to bring it into line with the<br>
model.<br>
<br>
This also has the added benefit of allowing certain schema migrations to<br>
be done online, while services are still running. This can significantly<br>
reduce downtime during deploys (a big concern for large deployments of<br>
Nova).<br>
<br>
There are some corner cases that do cause problems (renaming columns,<br>
changing column types, etc). Those can either remain as traditional<br>
migrations and/or discouraged.<br>
<br>
Data migrations would still remain with sqlalchemy-migrate/alembic, but<br>
there have some proposals about solving that problem too.<br>
<br>
JE<br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div><br></div>