Open Stack

Boris, what do you think?

Thanks,
Eugene.


On Tue, Jul 2, 2013 at 9:55 PM, Ben Nemec <openstack at nemebean.com> wrote:

> One small addition I would suggest is a step to remove the unused
> sqlalchemy-migrate code once this is all done.  That's my main concern with
> moving it to Oslo right now.
>
> Also, is this a formal blueprint(s)?  Seems like it should be.
>
> -Ben
>
> On 2013-07-02 12:50, Boris Pavlovic wrote:
>
>>         ##############################**##############################**
>> ###########
>>  Goal
>>
>>         ##############################**##############################**
>> ###########
>>
>> We should fix work with DB, unify it in all projects and use oslo code
>> for all common things.
>>
>> In more words:
>>
>> DB API
>>
>>  *) Fully cover by tests.
>>
>>  *) Run tests against all backends (now they are runed only against
>> sqlite).
>>
>>  *) Unique constraints (instead of select + insert)
>>  a) Provide unique constraints.
>>  b) Add missing unique constraints.
>>
>>  *) DB Archiving
>>  a) create shadow tables
>>  b) add tests that checks that shadow and main table are synced.
>>  c) add code that work with shadow tables.
>>
>>  *) DB API performance optimization
>>  a) Remove unused joins.
>>  b) 1 query instead of N (where it is possible).
>>  c) Add methods that could improve performance.
>>  d) Drop unused methods.
>>
>>  *) DB reconnect
>>  a) Don't break huge task if we lost connection for a moment. just
>> retry DB query.
>>
>>  *) DB Session cleanup
>>  a) do not use session parameter in public DB API methods.
>>  b) fix places where we are doing N queries in N transactions instead
>> of 1.
>>  c) get only data that is used (e.g. len(query.all()) =>
>> query.count()).
>>
>> ----
>>
>> DB Migrations
>>
>>  *) Test DB Migrations against all backends and real data.
>>
>>  *) Fix: DB schemas after Migrations should be same in different
>> backends
>>
>>  *) Fix: hidden bugs, that are caused by wrong migrations:
>>  a) fix indexes. e.g. 152 migration in Nova drop all Indexes that has
>> deleted column
>>  b) fix wrong types
>>  c) drop unused tables
>>
>>  *) Switch from sqlalchemy-migrate to something that is not death
>> (e.g. alembic).
>>
>> ----
>>
>> DB Models
>>
>>  *) Fix: Schema that is created by Models should be the same as after
>> migrations.
>>
>>  *) Fix: Unit tests should be runed on DB that was created by Models
>> not migrations.
>>
>>  *) Add test that checks that Models are synced with migrations.
>>
>> ----
>>
>> Oslo Code
>>
>>  *) Base Sqlalchemy Models.
>>
>>  *) Work around engine and session.
>>
>>  *) SqlAlchemy Utils - that helps us with migrations and tests.
>>
>>  *) Test migrations Base.
>>
>>  *) Use common test wrapper that allows us to run tests on different
>> backends.
>>
>>         ##############################**##############################**
>> ###########
>>  Implementation
>>
>>         ##############################**##############################**
>> ###########
>>
>>  This is really really huge task. And we are almost done with Nova=).
>>
>>  In OpenStack for such work there is only one approach ("baby steps"
>> development deriven). So we are making tons of patches that could be
>> easy reviewed. But there is also minuses in such approach. It is
>> pretty hard to track work on high level. And sometimes there are
>> misunderstand.
>>
>>  For example with oslo code. In few words at this moment we would like
>> to add (for some time) in oslo monkey patching for sqlalchemy-migrate.
>> And I got reasonable question from Doug Hellmann. Why? I answer
>> because of our "baby steps". But if you don't have a list of baby
>> steps it is pretty hard to understand why our baby steps need this
>> thing. And why we don't switch to alembic firstly. So I would like to
>> describe our Road Map and write list of "baby steps".
>>
>> -------
>>
>> OSLO
>>
>>  *) (Merged) Base code for Models and sqlalchemy engine (session)
>>
>>  *) (On review) Sqlalchemy utils that are used to:
>>  1. Fix bugs in sqlalchemy-migrate
>>  2. Base code for migrations that provides Unique Constraints.
>>  3. Utils for db.archiving helps us to create and check shadow tables.
>>
>>  *) (On review) Testtools wrapper
>>  We should have only one testtool wrapper in all projects. And this is
>> the one of base steps in task of running tests against all backends.
>>
>>  *) (On review) Test migrations base
>>  Base classes that provides us to test our migrations against all
>> backends on real data
>>
>>  *) (On review, not finished yet) DB Reconnect.
>>
>>  *) (Not finished) Test that checks that schemas and models are synced
>>
>> -------
>>
>> ${PROJECT_NAME}
>>
>> In different projects we could work absolutely simultaneously, and
>> first candidates are Glance and Cinder. But inside project we could
>> also work simultaneously. Here is the workflow:
>>
>>  1) (SYNC) Use base code for Models and sqlalchemy engines (from oslo)
>>
>>  2) (SYNC) Use test migrations base (from oslo)
>>
>>  3) (SYNC) Use SqlAlchemy utils (from oslo)
>>
>>  4) (1 patch) Switch to OSLO DB code
>>
>>  5) (1 patch) Remove ported test migrations
>>
>>  6) (1 Migration) Provide unique constraints (change type of "deleted"
>> column)
>>
>>  7) (1 Migration) Add shadow tables
>>  a) Create shadow tables
>>  b) Add test that checks that they are synced always
>>
>>  8) (N Migrations) UniqueConstraint/Session/**Optimization workflow:
>>  a) (1 patch) Add/Improve/Refactor tests for part of api (that is
>> connected with model)
>>  b) (1 patch) Fix session
>>  c) (1 patch) Optimize method
>>  d) if required (1 Migration) Add missing Unique Constraints.
>>
>>  9) (M Migrations) Sync Models with DB schemas (almost independent
>> from 8)
>>  a) (K pathces) Sync Models with MySQL Migrations
>>  (because they are different in different backneds)
>>  b) (Fix, L migrations) Fix migrations, that have bugs (e.g. 152
>> migraiton in Nova, drops all
>>  Indexes that have 'deleted' column).
>>  c) Add test for MySql
>>  d) (Fix, 2 migrations) Sync other backends schemas with MySql schema
>>  e) Add test for SQLite and Psql
>>  f) Rethink our models:
>>  *) Improve Indexes
>>  *) Use Enum where it is required
>>  *) Fix bugs if there are.
>>  *) Add missing default values in schema.
>>
>> 10) Test should be runed on schema that is created from Models, not
>> migrations (depend on 9)
>>
>> 11) Use alembic (or something else) instead of sqlalchemy-migrate
>> (after 9.e. it will be safe
>>  a) make test migrations independent from sqlalchemy-migrate
>>  b) (If alembic) Provide work with sqlite.
>>  c) Replace all migrations
>>
>> Main part of work is in 7, 8 and 10 points.
>>
>>         ##############################**##############################**
>> ###########
>>  Answer for Doug's question
>>          ##############################**##############################**
>> ###########
>>
>> Question:
>>  Why we should put in oslo slqlalchemy-migrate monkey patches, when we
>> are planing to switch to alembic?
>>
>> Answer:
>>  If we don't put in oslo sqlalchemy-migrate monkey patches. We won't
>> be able to work on 7 point at all until 8 and 10 points will be
>> implemented in every project. Also work around 8 point is not
>> finished, so we are not able to implement 10 points in any of project.
>> So this blocks almost all work in all projects. I think that these
>> 100-200 lines of code are not so big price for saving few cycles of
>> time.
>>
>>         ##############################**##############################**
>> ###########
>>  Current progress
>>
>>         ##############################**##############################**
>> ###########
>>
>> Nova
>>
>>  *) 1,4,6,7 - fully finished
>>
>>  *) 2,3,5 - waiting for oslo, but pretty simple to sync it
>>
>>  *) 8 - almost finished
>>  a) Few patched around tests (on review)
>>  b) 9 patches that add UniqueConstraints (on review) and few missing
>>  c) Few patches about session (on review) and few missing
>>
>>  *) 9 - tons of patches that syncs models with MySql on review. Also
>> we are working around tests that checks that schemas that are crated
>> by models and migrations are same. So it will be a lot of work here.
>>
>>  *) 10 - pretty simple (blocked by 9)
>>
>>  *) 11 - tons of work, candidate for I cycle.
>>
>>  *) Also here is work around run test on all backends but it is not
>> finished yet.
>>
>>  -------
>>
>> Cinder
>>
>>  *) 1,4 - finished
>>
>>  *) Some work around 8 (sessions and tests)
>>
>>  *) Other things are blocked by oslo
>>
>> -------
>>
>> Glance
>>
>>  *) 1 - finished
>>
>>  *) 4 - working on it
>>
>> Best regards,
>>
>> Boris Pavlović
>>
>> (irc boris-42)
>>
>>
>> ______________________________**_________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.**org <OpenStack-dev at lists.openstack.org>
>> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>
> ______________________________**_________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.**org <OpenStack-dev at lists.openstack.org>
> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130703/b17e5d8b/attachment.html>