<div dir="ltr">Boris, what do you think?<div><br></div><div>Thanks,</div><div>Eugene.</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jul 2, 2013 at 9:55 PM, Ben Nemec <span dir="ltr"><<a href="mailto:openstack@nemebean.com" target="_blank">openstack@nemebean.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">One small addition I would suggest is a step to remove the unused sqlalchemy-migrate code once this is all done. That's my main concern with moving it to Oslo right now.<br>
<br>
Also, is this a formal blueprint(s)? Seems like it should be.<br>
<br>
-Ben<br>
<br>
On 2013-07-02 12:50, Boris Pavlovic wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
##############################<u></u>##############################<u></u>###########<br>
Goal<div><div class="h5"><br>
##############################<u></u>##############################<u></u>###########<br>
<br>
We should fix work with DB, unify it in all projects and use oslo code<br>
for all common things.<br>
<br>
In more words:<br>
<br>
DB API<br>
<br>
*) Fully cover by tests.<br>
<br>
*) Run tests against all backends (now they are runed only against<br>
sqlite).<br>
<br>
*) Unique constraints (instead of select + insert)<br>
a) Provide unique constraints.<br>
b) Add missing unique constraints.<br>
<br>
*) DB Archiving<br>
a) create shadow tables<br>
b) add tests that checks that shadow and main table are synced.<br>
c) add code that work with shadow tables.<br>
<br>
*) DB API performance optimization<br>
a) Remove unused joins.<br>
b) 1 query instead of N (where it is possible).<br>
c) Add methods that could improve performance.<br>
d) Drop unused methods.<br>
<br>
*) DB reconnect<br>
a) Don't break huge task if we lost connection for a moment. just<br>
retry DB query.<br>
<br>
*) DB Session cleanup<br>
a) do not use session parameter in public DB API methods.<br>
b) fix places where we are doing N queries in N transactions instead<br>
of 1.<br>
c) get only data that is used (e.g. len(query.all()) =><br>
query.count()).<br>
<br>
----<br>
<br>
DB Migrations<br>
<br>
*) Test DB Migrations against all backends and real data.<br>
<br>
*) Fix: DB schemas after Migrations should be same in different<br>
backends<br>
<br>
*) Fix: hidden bugs, that are caused by wrong migrations:<br>
a) fix indexes. e.g. 152 migration in Nova drop all Indexes that has<br>
deleted column<br>
b) fix wrong types<br>
c) drop unused tables<br>
<br>
*) Switch from sqlalchemy-migrate to something that is not death<br>
(e.g. alembic).<br>
<br>
----<br>
<br>
DB Models<br>
<br>
*) Fix: Schema that is created by Models should be the same as after<br>
migrations.<br>
<br>
*) Fix: Unit tests should be runed on DB that was created by Models<br>
not migrations.<br>
<br>
*) Add test that checks that Models are synced with migrations.<br>
<br>
----<br>
<br>
Oslo Code<br>
<br>
*) Base Sqlalchemy Models.<br>
<br>
*) Work around engine and session.<br>
<br>
*) SqlAlchemy Utils - that helps us with migrations and tests.<br>
<br>
*) Test migrations Base.<br>
<br>
*) Use common test wrapper that allows us to run tests on different<br>
backends.<br>
<br>
##############################<u></u>##############################<u></u>###########<br></div></div>
Implementation<div><div class="h5"><br>
##############################<u></u>##############################<u></u>###########<br>
<br>
This is really really huge task. And we are almost done with Nova=).<br>
<br>
In OpenStack for such work there is only one approach ("baby steps"<br>
development deriven). So we are making tons of patches that could be<br>
easy reviewed. But there is also minuses in such approach. It is<br>
pretty hard to track work on high level. And sometimes there are<br>
misunderstand.<br>
<br>
For example with oslo code. In few words at this moment we would like<br>
to add (for some time) in oslo monkey patching for sqlalchemy-migrate.<br>
And I got reasonable question from Doug Hellmann. Why? I answer<br>
because of our "baby steps". But if you don't have a list of baby<br>
steps it is pretty hard to understand why our baby steps need this<br>
thing. And why we don't switch to alembic firstly. So I would like to<br>
describe our Road Map and write list of "baby steps".<br>
<br>
-------<br>
<br>
OSLO<br>
<br>
*) (Merged) Base code for Models and sqlalchemy engine (session)<br>
<br>
*) (On review) Sqlalchemy utils that are used to:<br>
1. Fix bugs in sqlalchemy-migrate<br>
2. Base code for migrations that provides Unique Constraints.<br>
3. Utils for db.archiving helps us to create and check shadow tables.<br>
<br>
*) (On review) Testtools wrapper<br>
We should have only one testtool wrapper in all projects. And this is<br>
the one of base steps in task of running tests against all backends.<br>
<br>
*) (On review) Test migrations base<br>
Base classes that provides us to test our migrations against all<br>
backends on real data<br>
<br>
*) (On review, not finished yet) DB Reconnect.<br>
<br>
*) (Not finished) Test that checks that schemas and models are synced<br>
<br>
-------<br>
<br>
${PROJECT_NAME}<br>
<br>
In different projects we could work absolutely simultaneously, and<br>
first candidates are Glance and Cinder. But inside project we could<br>
also work simultaneously. Here is the workflow:<br>
<br>
1) (SYNC) Use base code for Models and sqlalchemy engines (from oslo)<br>
<br>
2) (SYNC) Use test migrations base (from oslo)<br>
<br>
3) (SYNC) Use SqlAlchemy utils (from oslo)<br>
<br>
4) (1 patch) Switch to OSLO DB code<br>
<br>
5) (1 patch) Remove ported test migrations<br>
<br>
6) (1 Migration) Provide unique constraints (change type of "deleted"<br>
column)<br>
<br>
7) (1 Migration) Add shadow tables<br>
a) Create shadow tables<br>
b) Add test that checks that they are synced always<br>
<br>
8) (N Migrations) UniqueConstraint/Session/<u></u>Optimization workflow:<br>
a) (1 patch) Add/Improve/Refactor tests for part of api (that is<br>
connected with model)<br>
b) (1 patch) Fix session<br>
c) (1 patch) Optimize method<br>
d) if required (1 Migration) Add missing Unique Constraints.<br>
<br>
9) (M Migrations) Sync Models with DB schemas (almost independent<br>
from 8)<br>
a) (K pathces) Sync Models with MySQL Migrations<br>
(because they are different in different backneds)<br>
b) (Fix, L migrations) Fix migrations, that have bugs (e.g. 152<br>
migraiton in Nova, drops all<br>
Indexes that have 'deleted' column).<br>
c) Add test for MySql<br>
d) (Fix, 2 migrations) Sync other backends schemas with MySql schema<br>
e) Add test for SQLite and Psql<br>
f) Rethink our models:<br>
*) Improve Indexes<br>
*) Use Enum where it is required<br>
*) Fix bugs if there are.<br>
*) Add missing default values in schema.<br>
<br>
10) Test should be runed on schema that is created from Models, not<br>
migrations (depend on 9)<br>
<br>
11) Use alembic (or something else) instead of sqlalchemy-migrate<br>
(after 9.e. it will be safe<br>
a) make test migrations independent from sqlalchemy-migrate<br>
b) (If alembic) Provide work with sqlite.<br>
c) Replace all migrations<br>
<br>
Main part of work is in 7, 8 and 10 points.<br>
<br>
##############################<u></u>##############################<u></u>###########<br></div></div><div class="im">
Answer for Doug's question<br>
##############################<u></u>##############################<u></u>###########<br>
<br></div><div class="im">
Question:<br>
Why we should put in oslo slqlalchemy-migrate monkey patches, when we<br>
are planing to switch to alembic?<br>
<br>
Answer:<br>
If we don't put in oslo sqlalchemy-migrate monkey patches. We won't<br>
be able to work on 7 point at all until 8 and 10 points will be<br>
implemented in every project. Also work around 8 point is not<br>
finished, so we are not able to implement 10 points in any of project.<br>
So this blocks almost all work in all projects. I think that these<br>
100-200 lines of code are not so big price for saving few cycles of<br>
time.<br>
<br>
##############################<u></u>##############################<u></u>###########<br></div>
Current progress<div><div class="h5"><br>
##############################<u></u>##############################<u></u>###########<br>
<br>
Nova<br>
<br>
*) 1,4,6,7 - fully finished<br>
<br>
*) 2,3,5 - waiting for oslo, but pretty simple to sync it<br>
<br>
*) 8 - almost finished<br>
a) Few patched around tests (on review)<br>
b) 9 patches that add UniqueConstraints (on review) and few missing<br>
c) Few patches about session (on review) and few missing<br>
<br>
*) 9 - tons of patches that syncs models with MySql on review. Also<br>
we are working around tests that checks that schemas that are crated<br>
by models and migrations are same. So it will be a lot of work here.<br>
<br>
*) 10 - pretty simple (blocked by 9)<br>
<br>
*) 11 - tons of work, candidate for I cycle.<br>
<br>
*) Also here is work around run test on all backends but it is not<br>
finished yet.<br>
<br>
-------<br>
<br>
Cinder<br>
<br>
*) 1,4 - finished<br>
<br>
*) Some work around 8 (sessions and tests)<br>
<br>
*) Other things are blocked by oslo<br>
<br>
-------<br>
<br>
Glance<br>
<br>
*) 1 - finished<br>
<br>
*) 4 - working on it<br>
<br>
Best regards,<br>
<br>
Boris Pavlović<br>
<br>
(irc boris-42)<br>
<br>
<br></div></div>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</blockquote>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</blockquote></div><br></div>