<div dir="ltr">Vish, <div><br></div><div>Schema comparator could be written as on one simple bash script. So it won't take a lot of time to run it every time.</div><div>But all complex work around migrations will be much simpler in first approach (and there will be at least 2 times less work at all).<br>
</div><div><br></div><div>Also we shouldn't wait anything and could start work on it now (replacing one by one migration)</div><div><br></div><div>Best regards,</div><div>Boris Pavlovic</div><div><br></div><div><br></div>
<div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jul 17, 2013 at 1:06 AM, Vishvananda Ishaya <span dir="ltr"><<a href="mailto:vishvananda@gmail.com" target="_blank">vishvananda@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><br><div><div class="im"><div>On Jul 16, 2013, at 11:48 AM, Boris Pavlovic <<a href="mailto:boris@pavlovic.me" target="_blank">boris@pavlovic.me</a>> wrote:</div>
<br><blockquote type="cite"><div dir="ltr">David, <div><br></div><div>1. Dan Prince thing is equal useful and can help in both cases</div><div><br></div><div>2. We are not able to block all openstack for an half year to implement your plan</div>
<div>
<br></div><div>3. We are able only to convert only grizzly migrations not havana (because our customers should be able to switch from grizzly to havana)</div><div><br></div><div>4. We don't need to wait to make SQLAlchemy-migrate deprecated because script that allows in same way to use alembic and migrate is almost ready. </div>
<div><br></div><div>5. I just don't understand, we are spending tons of times to fix and unify work around DB in whole openstack. It is pretty complex and large task. And we should choose the way that is simpler:</div>
<div><br></div><div>Our approach: </div><div>1) Migrate N migrations to alembic step by step (this work is pretty simple) with tons of tests + Dan Prince method</div><div><br></div><div>Your approach:</div><div>1) Migrate N migrations to one big migrations (it is much more complex then our 1 step)</div>
<div>2) Replace one really huge migration to one Really huge migration in alembic (it is also more complex then our 1 step)</div><div><br></div><div>So instead on doing 1 long but simple step we are doing 2 complex. Could you explain me why?<br>
</div></div></blockquote><div><br></div></div><div>The testing is far simpler in the second approach. You only have to dump the output of the database between before 1, after 1, and after 2 and make sure that it is the same in each case. That means three versions that must be the same. In The first approach you have to verify that the database is exactly the same after each migration set, potentially 100 versions that must be in sync.</div>
<div><br></div><div>Vish</div><div><div class="h5"><br><blockquote type="cite"><div dir="ltr"><div>
</div><div><br></div><div>Best regards,</div><div>Boris Pavlovic</div><div><br></div><div><br></div><div><div class="gmail_extra"><div class="gmail_quote">On Tue, Jul 16, 2013 at 10:16 PM, David Ripton <span dir="ltr"><<a href="mailto:dripton@redhat.com" target="_blank">dripton@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>On 07/16/2013 01:58 PM, Boris Pavlovic wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
There is no "magic" and we have only 2 ways to end up with this problems<br>
and bugs that could be caused by "manually" migrations merging and tons<br>
of bugs in sqlalchemy-migrate.<br>
<br>
1) step by step (openstack method)<br>
There are special tests "test_migrations" that runs migrations on<br>
real data against all backends. So we should:<br>
<br>
a) improve this tests to checks all behaviors // there is a lot of<br>
hidden bugs<br>
b) replace migration (only one small migration) to alembic<br>
c) check that in all backends we made the same changes in schema<br>
d) Merge all old migrations in one using alembic (automatically).<br>
So it could be done in safe way.<br>
<br>
2.a) huge 2 steps<br>
1. Merge all migrations in one huge manually (drop all tests in test<br>
migrations)<br>
e.g. In Nova was patch <a href="https://review.openstack.org/#/c/35748/" target="_blank">https://review.openstack.org/#<u></u>/c/35748/</a><br>
I don't believe that there are no mistakes in this migration, and<br>
nobody is able to check it. // because of tons of hidden bugs in old<br>
migrations and sqla-migrate.<br>
2. Replace this migration in Alembic<br>
I don't believe that there will be way to check that there is no<br>
bugs<br>
<br>
2.b) suicide mode (1 big step)<br>
Merge and switch in one step=)<br>
</blockquote>
<br></div></div>
We have compacted migrations before, and there's a test document for how to verify that the big migration has exactly the same output as the series of small migrations. See <a href="https://wiki.openstack.org/wiki/Database_migration_testing" target="_blank">https://wiki.openstack.org/<u></u>wiki/Database_migration_<u></u>testing</a> Dan Prince is the expert on this.<br>
<br>
I think the right process is:<br>
<br>
1. Wait until the very beginning of Icehouse cycle. (But not after we have new migrations for Icehouse.)<br>
<br>
2. Compact all migrations into 2xx_havana.py (for SQLAlchemy-migrate)<br>
<br>
3. Test that it's perfect via above test plan plus whatever enhancements we think of.<br>
<br>
4. Manually convert 2xx_havana.py (for SQLAlchemy-migrate) into Alembic, and verify that it's still perfect.<br>
<br>
5. Deprecate the SQLAlchemy-migrate version and announce that new migrations should be in Alembic.<br>
<br>
#4 is hard work but not impossible. I have some old code that does 90% of the work, so we only have to do the other 90%.<span><font color="#888888"><br>
<br>
-- <br>
David Ripton Red Hat <a href="mailto:dripton@redhat.com" target="_blank">dripton@redhat.com</a></font></span><div><div><br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</div></div></blockquote></div><br></div></div></div>
_______________________________________________<br>OpenStack-dev mailing list<br><a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div></div></div><br></div><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>