Yogi just answered my question below -- In short, the ./manage.py test command didn't behave as I understood.<div><br></div><div>It simply tests the latest migration forwards & backwards, not the entire history of migrations in sequence.</div>
<div><br></div><div>Thanks, Yogi!</div><div><br></div><div>-Dolph<br><br><div class="gmail_quote">On Fri, Nov 4, 2011 at 10:37 AM, Dolph Mathews <span dir="ltr"><<a href="mailto:dolph.mathews@gmail.com">dolph.mathews@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Looking for some support from anyone with experience with sqlalchemy-migrate on the following review...<div><br></div>
<div><a href="https://review.openstack.org/#change,1200" target="_blank">https://review.openstack.org/#change,1200</a></div>
<div><br></div><div>See my Nov 1 comment -- sqlalchemy's built-in `./manage.py test` command fails, but you can test each migration individually (forward & backward) and they appear to work fine. It appears as though the test command is attempting to apply the third/last schema revision without running the prior migrations first? I'm hoping someone can tell me I'm doing something dumb...</div>
<div><br></div><div>Thanks!</div><div><br></div><div><font color="#888888">-Dolph</font><div><div></div><div class="h5"><br><br><div class="gmail_quote">On Fri, Oct 28, 2011 at 12:55 PM, Ziad Sawalha <span dir="ltr"><<a href="mailto:ziad.sawalha@rackspace.com" target="_blank">ziad.sawalha@rackspace.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0, 0, 0);font-size:14px;font-family:Calibri, sans-serif">
<div>Thanks, pvo. This looks like good. We'd like to follow what other projects do. Has there been any feedback on this BP? Will this be "the way" to handle things in OpenStack?</div>
<div><br>
</div>
<div>The easiest way for us to handle this now for schema migrations to Keystone is to follow Jesse's suggestion and use SQL Alchemy migrate_repo. We'll do that for schema changes currently in our branches and will work towards adopting the BP above.</div>
<div><br>
</div>
<div>Z</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<span>
<div style="font-family:Calibri;font-size:11pt;text-align:left;color:black;border-bottom:medium none;border-left:medium none;padding-bottom:0in;padding-left:0in;padding-right:0in;border-top:#b5c4df 1pt solid;border-right:medium none;padding-top:3pt">
<span style="font-weight:bold">From: </span>Paul Voccio <<a href="mailto:paul.voccio@rackspace.com" target="_blank">paul.voccio@rackspace.com</a>><br>
<span style="font-weight:bold">Date: </span>Tue, 25 Oct 2011 19:33:57 -0500<br>
<span style="font-weight:bold">To: </span>Brian Schott <<a href="mailto:brian.schott@nimbisservices.com" target="_blank">brian.schott@nimbisservices.com</a>>, Ziad Sawalha <<a href="mailto:ziad.sawalha@rackspace.com" target="_blank">ziad.sawalha@rackspace.com</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>" <<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Openstack] Handling Schema Changes in Keystone<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap:break-word;color:rgb(0, 0, 0);font-size:14px;font-family:Calibri, sans-serif">
<div>There is a BP out for how to handle continuious delivery and db upgrades within Nova. It does address some of the issues that keystone may face but feedback on the doc would be appreciated. </div>
<div><br>
</div>
<div><a href="https://blueprints.launchpad.net/nova/+spec/deployability-improvements" target="_blank">https://blueprints.launchpad.net/nova/+spec/deployability-improvements</a></div>
<div><br>
</div>
<div>pvo</div>
<div><br>
</div>
<span>
<div style="font-family:Calibri;font-size:11pt;text-align:left;color:black;border-bottom:medium none;border-left:medium none;padding-bottom:0in;padding-left:0in;padding-right:0in;border-top:#b5c4df 1pt solid;border-right:medium none;padding-top:3pt">
<span style="font-weight:bold">From: </span>Brian Schott <<a href="mailto:brian.schott@nimbisservices.com" target="_blank">brian.schott@nimbisservices.com</a>><br>
<span style="font-weight:bold">Date: </span>Tue, 25 Oct 2011 18:16:30 -0400<br>
<span style="font-weight:bold">To: </span>Ziad Sawalha <<a href="mailto:ziad.sawalha@rackspace.com" target="_blank">ziad.sawalha@rackspace.com</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>" <<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Openstack] Handling Schema Changes in Keystone<br>
</div>
<div><br>
</div>
<blockquote style="border-left:#b5c4df 5 solid;padding:0 0 0 5;margin:0 0 0 5">
<div>
<div style="word-wrap:break-word"><div>
It's probably the best approach that doesn't depend on the Django ORM (like South). Does anyone use the "experimental" command line migrate to generate the migrate script? I always did it by hand.
</div><div><a href="http://packages.python.org/sqlalchemy-migrate/" target="_blank">http://packages.python.org/sqlalchemy-migrate/</a><br>
<div><br>
</div><div>
<div>On the dev side, one of the big headaches in nova migrate_repo has been that the file numbering by hand meant that competing feature teams that needed to incorporate a schema change had to keep bumping the number every time a new version file got committed.
We talked about relaxing migrate_repo numbering rules so that you could create a 999_my_pending_change.py file that didn't break the version numbering adjacency checks. I don't know if that happened. Otherwise, every time a new file arrived in trunk, we
all had to manually renumber our development files from 055_x to 056_x... Not a big deal, but annoying when you pushed a branch up for review, then it broke because something else arrived in the same number slot.</div>
<div><br>
</div>
</div><div><div>On the deploy side, complicated table transforms don't always map well to all database backends and I don't think here is any unit testing with populated fixtures for data upgrade/downgrade. Don't know if this has been an issue in real deployments, but
the opportunity for sysadmin excellence is certainly there....</div>
<div><br>
</div>
</div><div><div>Brian </div>
<div><br>
</div>
<div>
<div>
<div>
<div style="word-wrap:break-word">
<div style="word-wrap:break-word">
<div style="word-wrap:break-word">
<div style="word-wrap:break-word;font-family:Helvetica;font-size:12px">
<div>-------------------------------------------------</div>
<div>Brian Schott, CTO</div>
<div>Nimbis Services, Inc.</div>
<div><a href="mailto:brian.schott@nimbisservices.com" target="_blank">brian.schott@nimbisservices.com</a></div>
<div>ph: <a href="tel:443-274-6064" value="+14432746064" target="_blank">443-274-6064</a> fx: <a href="tel:443-274-6060" value="+14432746060" target="_blank">443-274-6060</a></div>
<div><br>
</div>
</div>
<br>
</div>
<br>
</div>
<br>
</div>
<br>
<br>
</div>
<br>
<div>
<div>On Oct 25, 2011, at 5:45 PM, Ziad Sawalha wrote:</div>
<br>
<blockquote type="cite">
<div style="word-wrap:break-word">
<div style="color:rgb(0, 0, 0);font-family:Calibri, sans-serif;font-size:14px">
Our schema right now is auto generated from the model using sqlalchemy. Whenever we change the model, the generated schema is different for new installations but this does not address existing deployments.</div>
<div style="color:rgb(0, 0, 0);font-family:Calibri, sans-serif;font-size:14px">
<br>
</div>
<div style="color:rgb(0, 0, 0);font-family:Calibri, sans-serif;font-size:14px">
Looking for feedback on how to handle this better:</div>
<div style="color:rgb(0, 0, 0);font-family:Calibri, sans-serif;font-size:14px">
anotherjesses offered: <span style="color:rgb(20, 20, 20);line-height:14px;font-size:medium;font-family:'Lucida Grande'"><a href="https://github.com/openstack/nova/tree/master/nova/db/sqlalchemy/migrate_repo" title="https://github.com/openstack/nova/tree/master/nova/db/sqlalchemy/migrate_repo" style="word-wrap:break-word;color:rgb(0, 51, 204);text-decoration:underline" target="_blank">https://github.com/openstack/nova/tree/master/nova/db/sqlalchemy/migrate_repo</a></span></div>
<div><span style="font-size:14px;font-family:Calibri, sans-serif"><br>
</span></div>
<div><span style="font-size:14px;font-family:Calibri, sans-serif">Questions:</span></div>
<div><span style="font-size:14px;font-family:Calibri, sans-serif">- Has anyone used this on the dev side?</span></div>
<div><span style="font-size:14px;font-family:Calibri, sans-serif">- Has anyone used this on the deployment/ops side?</span></div>
<div><span style="font-size:14px;font-family:Calibri, sans-serif"><br>
</span></div>
<div><span style="font-size:14px;font-family:Calibri, sans-serif">Would love to hear from you how you started it (we have multiple versions of our schema out there, so where do we start) and what was the experience updating versions.</span></div>
<div><span style="font-size:14px;font-family:Calibri, sans-serif"><br>
</span></div>
<div><span style="font-size:14px;font-family:Calibri, sans-serif">Regards,</span></div>
<div><span style="font-size:14px;font-family:Calibri, sans-serif"><br>
</span></div>
<div><font face="Calibri,sans-serif">Ziad</font></div>
</div>
_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div></div>
</div>
</div><div>
_______________________________________________ Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">
https://launchpad.net/~openstack</a> Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">
openstack@lists.launchpad.net</a> Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">
https://launchpad.net/~openstack</a> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">
https://help.launchpad.net/ListHelp</a> </div></blockquote>
</span></div>
</div>
</span>
</div>
<br>_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br></blockquote></div><br></div></div></div>
</blockquote></div><br></div>