Open Stack

One small addition I would suggest is a step to remove the unused 
sqlalchemy-migrate code once this is all done.  That's my main concern 
with moving it to Oslo right now.

Also, is this a formal blueprint(s)?  Seems like it should be.

-Ben

On 2013-07-02 12:50, Boris Pavlovic wrote:
>  
>  
>  
>  
>  
>  
>  
>  
> #######################################################################
>  Goal
>  
>  
>  
>  
>  
>  
>  
>  
> #######################################################################
> 
> We should fix work with DB, unify it in all projects and use oslo code
> for all common things.
> 
> In more words:
> 
> DB API
> 
>  *) Fully cover by tests.
> 
>  *) Run tests against all backends (now they are runed only against
> sqlite).
> 
>  *) Unique constraints (instead of select + insert)
>  a) Provide unique constraints.
>  b) Add missing unique constraints.
> 
>  *) DB Archiving
>  a) create shadow tables
>  b) add tests that checks that shadow and main table are synced.
>  c) add code that work with shadow tables.
> 
>  *) DB API performance optimization
>  a) Remove unused joins.
>  b) 1 query instead of N (where it is possible).
>  c) Add methods that could improve performance.
>  d) Drop unused methods.
> 
>  *) DB reconnect
>  a) Don't break huge task if we lost connection for a moment. just
> retry DB query.
> 
>  *) DB Session cleanup
>  a) do not use session parameter in public DB API methods.
>  b) fix places where we are doing N queries in N transactions instead
> of 1.
>  c) get only data that is used (e.g. len(query.all()) =>
> query.count()).
> 
> ----
> 
> DB Migrations
> 
>  *) Test DB Migrations against all backends and real data.
> 
>  *) Fix: DB schemas after Migrations should be same in different
> backends
> 
>  *) Fix: hidden bugs, that are caused by wrong migrations:
>  a) fix indexes. e.g. 152 migration in Nova drop all Indexes that has
> deleted column
>  b) fix wrong types
>  c) drop unused tables
> 
>  *) Switch from sqlalchemy-migrate to something that is not death
> (e.g. alembic).
> 
> ----
> 
> DB Models
> 
>  *) Fix: Schema that is created by Models should be the same as after
> migrations.
> 
>  *) Fix: Unit tests should be runed on DB that was created by Models
> not migrations.
> 
>  *) Add test that checks that Models are synced with migrations.
> 
> ----
> 
> Oslo Code
> 
>  *) Base Sqlalchemy Models.
> 
>  *) Work around engine and session.
> 
>  *) SqlAlchemy Utils - that helps us with migrations and tests.
> 
>  *) Test migrations Base.
> 
>  *) Use common test wrapper that allows us to run tests on different
> backends.
> 
>  
>  
>  
>  
>  
>  
>  
>  
> #######################################################################
>  Implementation
>  
>  
>  
>  
>  
>  
>  
>  
> #######################################################################
> 
>  This is really really huge task. And we are almost done with Nova=).
> 
>  In OpenStack for such work there is only one approach ("baby steps"
> development deriven). So we are making tons of patches that could be
> easy reviewed. But there is also minuses in such approach. It is
> pretty hard to track work on high level. And sometimes there are
> misunderstand.
> 
>  For example with oslo code. In few words at this moment we would like
> to add (for some time) in oslo monkey patching for sqlalchemy-migrate.
> And I got reasonable question from Doug Hellmann. Why? I answer
> because of our "baby steps". But if you don't have a list of baby
> steps it is pretty hard to understand why our baby steps need this
> thing. And why we don't switch to alembic firstly. So I would like to
> describe our Road Map and write list of "baby steps".
> 
> -------
> 
> OSLO
> 
>  *) (Merged) Base code for Models and sqlalchemy engine (session)
> 
>  *) (On review) Sqlalchemy utils that are used to:
>  1. Fix bugs in sqlalchemy-migrate
>  2. Base code for migrations that provides Unique Constraints.
>  3. Utils for db.archiving helps us to create and check shadow tables.
> 
>  *) (On review) Testtools wrapper
>  We should have only one testtool wrapper in all projects. And this is
> the one of base steps in task of running tests against all backends.
> 
>  *) (On review) Test migrations base
>  Base classes that provides us to test our migrations against all
> backends on real data
> 
>  *) (On review, not finished yet) DB Reconnect.
> 
>  *) (Not finished) Test that checks that schemas and models are synced
> 
> -------
> 
> ${PROJECT_NAME}
> 
> In different projects we could work absolutely simultaneously, and
> first candidates are Glance and Cinder. But inside project we could
> also work simultaneously. Here is the workflow:
> 
>  1) (SYNC) Use base code for Models and sqlalchemy engines (from oslo)
> 
>  2) (SYNC) Use test migrations base (from oslo)
> 
>  3) (SYNC) Use SqlAlchemy utils (from oslo)
> 
>  4) (1 patch) Switch to OSLO DB code
> 
>  5) (1 patch) Remove ported test migrations
> 
>  6) (1 Migration) Provide unique constraints (change type of "deleted"
> column)
> 
>  7) (1 Migration) Add shadow tables
>  a) Create shadow tables
>  b) Add test that checks that they are synced always
> 
>  8) (N Migrations) UniqueConstraint/Session/Optimization workflow:
>  a) (1 patch) Add/Improve/Refactor tests for part of api (that is
> connected with model)
>  b) (1 patch) Fix session
>  c) (1 patch) Optimize method
>  d) if required (1 Migration) Add missing Unique Constraints.
> 
>  9) (M Migrations) Sync Models with DB schemas (almost independent
> from 8)
>  a) (K pathces) Sync Models with MySQL Migrations
>  (because they are different in different backneds)
>  b) (Fix, L migrations) Fix migrations, that have bugs (e.g. 152
> migraiton in Nova, drops all
>  Indexes that have 'deleted' column).
>  c) Add test for MySql
>  d) (Fix, 2 migrations) Sync other backends schemas with MySql schema
>  e) Add test for SQLite and Psql
>  f) Rethink our models:
>  *) Improve Indexes
>  *) Use Enum where it is required
>  *) Fix bugs if there are.
>  *) Add missing default values in schema.
> 
> 10) Test should be runed on schema that is created from Models, not
> migrations (depend on 9)
> 
> 11) Use alembic (or something else) instead of sqlalchemy-migrate
> (after 9.e. it will be safe
>  a) make test migrations independent from sqlalchemy-migrate
>  b) (If alembic) Provide work with sqlite.
>  c) Replace all migrations
> 
> Main part of work is in 7, 8 and 10 points.
> 
>  
>  
>  
>  
>  
>  
>  
>  
> #######################################################################
>  Answer for Doug's question
>  
>  
>  
>  
>  
>  
>  
>  
>  
> #######################################################################
> 
> Question:
>  Why we should put in oslo slqlalchemy-migrate monkey patches, when we
> are planing to switch to alembic?
> 
> Answer:
>  If we don't put in oslo sqlalchemy-migrate monkey patches. We won't
> be able to work on 7 point at all until 8 and 10 points will be
> implemented in every project. Also work around 8 point is not
> finished, so we are not able to implement 10 points in any of project.
> So this blocks almost all work in all projects. I think that these
> 100-200 lines of code are not so big price for saving few cycles of
> time.
> 
>  
>  
>  
>  
>  
>  
>  
>  
> #######################################################################
>  Current progress
>  
>  
>  
>  
>  
>  
>  
>  
> #######################################################################
> 
> Nova
> 
>  *) 1,4,6,7 - fully finished
> 
>  *) 2,3,5 - waiting for oslo, but pretty simple to sync it
> 
>  *) 8 - almost finished
>  a) Few patched around tests (on review)
>  b) 9 patches that add UniqueConstraints (on review) and few missing
>  c) Few patches about session (on review) and few missing
> 
>  *) 9 - tons of patches that syncs models with MySql on review. Also
> we are working around tests that checks that schemas that are crated
> by models and migrations are same. So it will be a lot of work here.
> 
>  *) 10 - pretty simple (blocked by 9)
> 
>  *) 11 - tons of work, candidate for I cycle.
> 
>  *) Also here is work around run test on all backends but it is not
> finished yet.
> 
>  -------
> 
> Cinder
> 
>  *) 1,4 - finished
> 
>  *) Some work around 8 (sessions and tests)
> 
>  *) Other things are blocked by oslo
> 
> -------
> 
> Glance
> 
>  *) 1 - finished
> 
>  *) 4 - working on it
> 
> Best regards,
> 
> Boris Pavlović
> 
> (irc boris-42)
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev