Open Stack

On 07/02/2013 10:50 AM, Boris Pavlovic wrote:
> ###############################################################################
>                                                                 Goal
> ###############################################################################
> 
> We should fix work with DB, unify it in all projects and use oslo code
> for all common things.

Just wanted to say a quick word that isn't about migrations...

Thank you. This is all great, and I'm thrilled someone is taking on the
task of fixing what is probably one of OpenStack's biggest nightmares.

> In more words:
> 
> DB API
> 
>   *) Fully cover by tests.
> 
>   *) Run tests against all backends (now they are runed only against
> sqlite).
> 
>   *) Unique constraints (instead of select + insert)
>      a) Provide unique constraints.
>      b) Add missing unique constraints.
> 
>   *) DB Archiving
>      a) create shadow tables
>      b) add tests that checks that shadow and main table are synced.
>      c) add code that work with shadow tables.
> 
>   *) DB API performance optimization
>     a) Remove unused joins..
>     b) 1 query instead of N (where it is possible).
>     c) Add methods that could improve performance.
>     d) Drop unused methods.
> 
>   *) DB reconnect
>     a) Don’t break huge task if we lost connection for a moment.. just
> retry DB query.
> 
>   *) DB Session cleanup
>     a) do not use session parameter in public DB API methods.
>     b) fix places where we are doing N queries in N transactions instead
> of 1.
>     c) get only data that is used (e.g. len(query.all()) => query.count()).
> 
> ----
> 
> DB Migrations
> 
>   *) Test DB Migrations against all backends and real data.
> 
>   *) Fix: DB schemas after Migrations should be same in different backends
> 
>   *) Fix: hidden bugs, that are caused by wrong migrations:
>      a) fix indexes. e.g. 152 migration in Nova drop all Indexes that
> has deleted column
>      b) fix wrong types
>      c) drop unused tables
> 
>   *) Switch from sqlalchemy-migrate to something that is not death (e.g.
> alembic).
> 
> ----
> 
> DB Models
> 
>   *) Fix: Schema that is created by Models should be the same as after
> migrations.
>  
>   *) Fix: Unit tests should be runed on DB that was created by Models
> not migrations.
> 
>   *) Add test that checks that Models are synced with migrations.
> 
> ----
> 
> Oslo Code
> 
>   *) Base Sqlalchemy Models.
> 
>   *) Work around engine and session.
> 
>   *) SqlAlchemy Utils - that helps us with migrations and tests.
> 
>   *) Test migrations Base.
> 
>   *) Use common test wrapper that allows us to run tests on different
> backends.
> 
> 
> ###############################################################################
>                                                        Implementation
> ###############################################################################
> 
>   This is really really huge task. And we are almost done with Nova=).
> 
>   In OpenStack for such work there is only one approach (“baby steps”
> development deriven). So we are making tons of patches that could be
> easy reviewed. But there is also minuses in such approach. It is pretty
> hard to track work on high level. And sometimes there are misunderstand.
>  
>   For example with oslo code. In few words at this moment we would like
> to add (for some time) in oslo monkey patching for sqlalchemy-migrate.
> And I got reasonable question from Doug Hellmann. Why? I answer because
> of our “baby steps”. But if you don’t have a list of baby steps it is
> pretty hard to understand why our baby steps need this thing. And why we
> don’t switch to alembic firstly. So I would like to describe our Road
> Map and write list of "baby steps".
> 
> 
> -------
> 
> OSLO
> 
>   *) (Merged) Base code for Models and sqlalchemy engine (session)
> 
>   *) (On review) Sqlalchemy utils that are used to:
>       1. Fix bugs in sqlalchemy-migrate
>       2. Base code for migrations that provides Unique Constraints.
>       3. Utils for db.archiving helps us to create and check shadow tables.
> 
>   *) (On review) Testtools wrapper
>        We should have only one testtool wrapper in all projects. And
> this is the one of base steps in task of running tests against all backends.
> 
>   *) (On review) Test migrations base
>        Base classes that provides us to test our migrations against all
> backends on real data
> 
>   *) (On review, not finished yet) DB Reconnect.  
> 
>   *) (Not finished) Test that checks that schemas and models are synced
> 
> -------
> 
> ${PROJECT_NAME}
> 
> 
> In different projects we could work absolutely simultaneously, and first
> candidates are Glance and Cinder. But inside project we could also work
> simultaneously. Here is the workflow:
> 
> 
>   1) (SYNC) Use base code for Models and sqlalchemy engines (from oslo)
> 
>   2) (SYNC) Use test migrations base (from oslo)
> 
>   3) (SYNC) Use SqlAlchemy utils (from oslo)
> 
>   4) (1 patch) Switch to OSLO DB code
> 
>   5) (1 patch) Remove ported test migrations
> 
>   6) (1 Migration) Provide unique constraints (change type of “deleted”
> column)
> 
>   7) (1 Migration) Add shadow tables
>     a) Create shadow tables
>     b) Add test that checks that they are synced always
> 
>   8) (N Migrations) UniqueConstraint/Session/Optimization workflow:
>     a) (1 patch) Add/Improve/Refactor tests for part of api (that is
> connected with model)
>     b) (1 patch) Fix session
>     c) (1 patch)  Optimize method
>     d) if required (1 Migration) Add missing Unique Constraints.
> 
>   9) (M Migrations) Sync Models with DB schemas (almost independent from 8)
>     a) (K pathces) Sync Models with MySQL Migrations
>         (because they are different in different backneds)
>     b) (Fix, L migrations) Fix migrations, that have bugs (e.g. 152
> migraiton in Nova, drops all
>          Indexes that have ‘deleted’ column).
>     c) Add test for MySql
>     d) (Fix, 2 migrations) Sync other backends schemas with MySql schema
>     e) Add test for SQLite and Psql
>     f) Rethink our models:
>        *) Improve Indexes
>        *) Use Enum where it is required
>        *) Fix bugs if there are.
>        *) Add missing default values in schema. 
> 
> 10) Test should be runed on schema that is created from Models, not
> migrations (depend on 9)
> 
> 11) Use alembic (or something else) instead of sqlalchemy-migrate (after
> 9.e. it will be safe
>   a) make test migrations independent from sqlalchemy-migrate
>   b) (If alembic) Provide work with sqlite.
>   c) Replace all migrations
> 
> Main part of work is in 7, 8 and 10 points.
> 
> 
> ###############################################################################
>                                            Answer for Doug's question
> ###############################################################################
> 
> Question:
>   Why we should put in oslo slqlalchemy-migrate monkey patches, when we
> are planing to switch to alembic?
> 
> Answer:
>    If we don’t put in oslo sqlalchemy-migrate monkey patches. We won't
> be able to work on 7 point at all until 8 and 10 points will be
> implemented in every project. Also work around 8 point is not finished,
> so we are not able to implement 10 points in any of project. So this
> blocks almost all work in all projects. I think that these 100-200 lines
> of code are not so big price for saving few cycles of time.
> 
> 
> 
> ###############################################################################
>                                            Current progress
> ###############################################################################
> 
> 
> Nova
> 
>   *) 1,4,6,7 - fully finished
> 
>   *) 2,3,5 - waiting for oslo, but pretty simple to sync it
> 
>   *) 8 - almost finished
>     a) Few patched around tests (on review)
>     b) 9 patches that add UniqueConstraints (on review) and few missing
>     c) Few patches about session (on review) and few missing
> 
>   *) 9 - tons of patches that syncs models with MySql on review. Also we
> are working around tests that checks that schemas that are crated by
> models and migrations are same. So it will be a lot of work here.
> 
>   *) 10 - pretty simple (blocked by 9)
> 
>   *) 11 - tons of work, candidate for I cycle.
>  
>   *) Also here is work around run test on all backends but it is not
> finished yet.
> 
> -------
> 
> Cinder
> 
>    *) 1,4 - finished
> 
>    *) Some work around 8 (sessions and tests)
>  
>    *) Other things are blocked by oslo
> 
> -------
> 
> Glance
> 
>    *) 1 - finished
> 
>    *) 4 - working on it
> 
> 
> 
> Best regards,
> Boris Pavlović
> 
> (irc boris-42)
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>