<div dir="ltr">Hi Monty, <div><br></div><div><span style="font-family:arial,sans-serif;font-size:13px">>>> I think if you're working on a non-alembic plan and boris is working on</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">>>> an alembic plan, then something is going to be unhappy in the</span><br style="font-family:arial,sans-serif;font-size:13px"><span style="font-family:arial,sans-serif;font-size:13px">>>> not-too-distant future. Can we get alignment on this?</span><br>
</div><div><br></div><div><br></div><div style>As I said before, we are preparing our DB code to move from sqlalchemy-migrate to something another.</div><div style>There will be a tons of work before we will be able to rewrite or migration scripts to alembic or something else.</div>
<div style><br></div><div style>And we are not sure that we would like to use alembic=)</div><div style><br></div><div style><br></div><div style>Best regards,</div><div style>Boris Pavlovic</div><div style><br></div></div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jul 3, 2013 at 9:30 PM, Monty Taylor <span dir="ltr"><<a href="mailto:mordred@inaugust.com" target="_blank">mordred@inaugust.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im"><br>
<br>
On 07/02/2013 10:50 AM, Boris Pavlovic wrote:<br>
> ###############################################################################<br>
> Goal<br>
> ###############################################################################<br>
><br>
> We should fix work with DB, unify it in all projects and use oslo code<br>
> for all common things.<br>
<br>
</div>Just wanted to say a quick word that isn't about migrations...<br>
<br>
Thank you. This is all great, and I'm thrilled someone is taking on the<br>
task of fixing what is probably one of OpenStack's biggest nightmares.<br>
<div class="im"><br>
> In more words:<br>
><br>
> DB API<br>
><br>
> *) Fully cover by tests.<br>
><br>
> *) Run tests against all backends (now they are runed only against<br>
> sqlite).<br>
><br>
> *) Unique constraints (instead of select + insert)<br>
> a) Provide unique constraints.<br>
> b) Add missing unique constraints.<br>
><br>
> *) DB Archiving<br>
> a) create shadow tables<br>
> b) add tests that checks that shadow and main table are synced.<br>
> c) add code that work with shadow tables.<br>
><br>
> *) DB API performance optimization<br>
</div>> a) Remove unused joins..<br>
<div class="im">> b) 1 query instead of N (where it is possible).<br>
> c) Add methods that could improve performance.<br>
> d) Drop unused methods.<br>
><br>
> *) DB reconnect<br>
</div>> a) Don’t break huge task if we lost connection for a moment.. just<br>
<div class="HOEnZb"><div class="h5">> retry DB query.<br>
><br>
> *) DB Session cleanup<br>
> a) do not use session parameter in public DB API methods.<br>
> b) fix places where we are doing N queries in N transactions instead<br>
> of 1.<br>
> c) get only data that is used (e.g. len(query.all()) => query.count()).<br>
><br>
> ----<br>
><br>
> DB Migrations<br>
><br>
> *) Test DB Migrations against all backends and real data.<br>
><br>
> *) Fix: DB schemas after Migrations should be same in different backends<br>
><br>
> *) Fix: hidden bugs, that are caused by wrong migrations:<br>
> a) fix indexes. e.g. 152 migration in Nova drop all Indexes that<br>
> has deleted column<br>
> b) fix wrong types<br>
> c) drop unused tables<br>
><br>
> *) Switch from sqlalchemy-migrate to something that is not death (e.g.<br>
> alembic).<br>
><br>
> ----<br>
><br>
> DB Models<br>
><br>
> *) Fix: Schema that is created by Models should be the same as after<br>
> migrations.<br>
><br>
> *) Fix: Unit tests should be runed on DB that was created by Models<br>
> not migrations.<br>
><br>
> *) Add test that checks that Models are synced with migrations.<br>
><br>
> ----<br>
><br>
> Oslo Code<br>
><br>
> *) Base Sqlalchemy Models.<br>
><br>
> *) Work around engine and session.<br>
><br>
> *) SqlAlchemy Utils - that helps us with migrations and tests.<br>
><br>
> *) Test migrations Base.<br>
><br>
> *) Use common test wrapper that allows us to run tests on different<br>
> backends.<br>
><br>
><br>
> ###############################################################################<br>
> Implementation<br>
> ###############################################################################<br>
><br>
> This is really really huge task. And we are almost done with Nova=).<br>
><br>
> In OpenStack for such work there is only one approach (“baby steps”<br>
> development deriven). So we are making tons of patches that could be<br>
> easy reviewed. But there is also minuses in such approach. It is pretty<br>
> hard to track work on high level. And sometimes there are misunderstand.<br>
><br>
> For example with oslo code. In few words at this moment we would like<br>
> to add (for some time) in oslo monkey patching for sqlalchemy-migrate.<br>
> And I got reasonable question from Doug Hellmann. Why? I answer because<br>
> of our “baby steps”. But if you don’t have a list of baby steps it is<br>
> pretty hard to understand why our baby steps need this thing. And why we<br>
> don’t switch to alembic firstly. So I would like to describe our Road<br>
> Map and write list of "baby steps".<br>
><br>
><br>
> -------<br>
><br>
> OSLO<br>
><br>
> *) (Merged) Base code for Models and sqlalchemy engine (session)<br>
><br>
> *) (On review) Sqlalchemy utils that are used to:<br>
> 1. Fix bugs in sqlalchemy-migrate<br>
> 2. Base code for migrations that provides Unique Constraints.<br>
> 3. Utils for db.archiving helps us to create and check shadow tables.<br>
><br>
> *) (On review) Testtools wrapper<br>
> We should have only one testtool wrapper in all projects. And<br>
> this is the one of base steps in task of running tests against all backends.<br>
><br>
> *) (On review) Test migrations base<br>
> Base classes that provides us to test our migrations against all<br>
> backends on real data<br>
><br>
> *) (On review, not finished yet) DB Reconnect.<br>
><br>
> *) (Not finished) Test that checks that schemas and models are synced<br>
><br>
> -------<br>
><br>
> ${PROJECT_NAME}<br>
><br>
><br>
> In different projects we could work absolutely simultaneously, and first<br>
> candidates are Glance and Cinder. But inside project we could also work<br>
> simultaneously. Here is the workflow:<br>
><br>
><br>
> 1) (SYNC) Use base code for Models and sqlalchemy engines (from oslo)<br>
><br>
> 2) (SYNC) Use test migrations base (from oslo)<br>
><br>
> 3) (SYNC) Use SqlAlchemy utils (from oslo)<br>
><br>
> 4) (1 patch) Switch to OSLO DB code<br>
><br>
> 5) (1 patch) Remove ported test migrations<br>
><br>
> 6) (1 Migration) Provide unique constraints (change type of “deleted”<br>
> column)<br>
><br>
> 7) (1 Migration) Add shadow tables<br>
> a) Create shadow tables<br>
> b) Add test that checks that they are synced always<br>
><br>
> 8) (N Migrations) UniqueConstraint/Session/Optimization workflow:<br>
> a) (1 patch) Add/Improve/Refactor tests for part of api (that is<br>
> connected with model)<br>
> b) (1 patch) Fix session<br>
> c) (1 patch) Optimize method<br>
> d) if required (1 Migration) Add missing Unique Constraints.<br>
><br>
> 9) (M Migrations) Sync Models with DB schemas (almost independent from 8)<br>
> a) (K pathces) Sync Models with MySQL Migrations<br>
> (because they are different in different backneds)<br>
> b) (Fix, L migrations) Fix migrations, that have bugs (e.g. 152<br>
> migraiton in Nova, drops all<br>
> Indexes that have ‘deleted’ column).<br>
> c) Add test for MySql<br>
> d) (Fix, 2 migrations) Sync other backends schemas with MySql schema<br>
> e) Add test for SQLite and Psql<br>
> f) Rethink our models:<br>
> *) Improve Indexes<br>
> *) Use Enum where it is required<br>
> *) Fix bugs if there are.<br>
> *) Add missing default values in schema.<br>
><br>
> 10) Test should be runed on schema that is created from Models, not<br>
> migrations (depend on 9)<br>
><br>
> 11) Use alembic (or something else) instead of sqlalchemy-migrate (after<br>
> 9.e. it will be safe<br>
> a) make test migrations independent from sqlalchemy-migrate<br>
> b) (If alembic) Provide work with sqlite.<br>
> c) Replace all migrations<br>
><br>
> Main part of work is in 7, 8 and 10 points.<br>
><br>
><br>
> ###############################################################################<br>
> Answer for Doug's question<br>
> ###############################################################################<br>
><br>
> Question:<br>
> Why we should put in oslo slqlalchemy-migrate monkey patches, when we<br>
> are planing to switch to alembic?<br>
><br>
> Answer:<br>
> If we don’t put in oslo sqlalchemy-migrate monkey patches. We won't<br>
> be able to work on 7 point at all until 8 and 10 points will be<br>
> implemented in every project. Also work around 8 point is not finished,<br>
> so we are not able to implement 10 points in any of project. So this<br>
> blocks almost all work in all projects. I think that these 100-200 lines<br>
> of code are not so big price for saving few cycles of time.<br>
><br>
><br>
><br>
> ###############################################################################<br>
> Current progress<br>
> ###############################################################################<br>
><br>
><br>
> Nova<br>
><br>
> *) 1,4,6,7 - fully finished<br>
><br>
> *) 2,3,5 - waiting for oslo, but pretty simple to sync it<br>
><br>
> *) 8 - almost finished<br>
> a) Few patched around tests (on review)<br>
> b) 9 patches that add UniqueConstraints (on review) and few missing<br>
> c) Few patches about session (on review) and few missing<br>
><br>
> *) 9 - tons of patches that syncs models with MySql on review. Also we<br>
> are working around tests that checks that schemas that are crated by<br>
> models and migrations are same. So it will be a lot of work here.<br>
><br>
> *) 10 - pretty simple (blocked by 9)<br>
><br>
> *) 11 - tons of work, candidate for I cycle.<br>
><br>
> *) Also here is work around run test on all backends but it is not<br>
> finished yet.<br>
><br>
> -------<br>
><br>
> Cinder<br>
><br>
> *) 1,4 - finished<br>
><br>
> *) Some work around 8 (sessions and tests)<br>
><br>
> *) Other things are blocked by oslo<br>
><br>
> -------<br>
><br>
> Glance<br>
><br>
> *) 1 - finished<br>
><br>
> *) 4 - working on it<br>
><br>
><br>
><br>
> Best regards,<br>
> Boris Pavloviæ<br>
><br>
> (irc boris-42)<br>
><br>
><br>
><br>
</div></div><div class="HOEnZb"><div class="h5">> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>