Open Stack

Hi Monty,

>>> I think if you're working on a non-alembic plan and boris is working on
>>> an alembic plan, then something is going to be unhappy in the
>>> not-too-distant future. Can we get alignment on this?


As I said before, we are preparing our DB code to move from
sqlalchemy-migrate to something another.
There will be a tons of work before we will be able to rewrite or migration
scripts to alembic or something else.

And we are not sure that we would like to use alembic=)


Best regards,
Boris Pavlovic



On Wed, Jul 3, 2013 at 9:30 PM, Monty Taylor <mordred at inaugust.com> wrote:

>
>
> On 07/02/2013 10:50 AM, Boris Pavlovic wrote:
> >
> ###############################################################################
> >                                                                 Goal
> >
> ###############################################################################
> >
> > We should fix work with DB, unify it in all projects and use oslo code
> > for all common things.
>
> Just wanted to say a quick word that isn't about migrations...
>
> Thank you. This is all great, and I'm thrilled someone is taking on the
> task of fixing what is probably one of OpenStack's biggest nightmares.
>
> > In more words:
> >
> > DB API
> >
> >   *) Fully cover by tests.
> >
> >   *) Run tests against all backends (now they are runed only against
> > sqlite).
> >
> >   *) Unique constraints (instead of select + insert)
> >      a) Provide unique constraints.
> >      b) Add missing unique constraints.
> >
> >   *) DB Archiving
> >      a) create shadow tables
> >      b) add tests that checks that shadow and main table are synced.
> >      c) add code that work with shadow tables.
> >
> >   *) DB API performance optimization
> >     a) Remove unused joins..
> >     b) 1 query instead of N (where it is possible).
> >     c) Add methods that could improve performance.
> >     d) Drop unused methods.
> >
> >   *) DB reconnect
> >     a) Don’t break huge task if we lost connection for a moment.. just
> > retry DB query.
> >
> >   *) DB Session cleanup
> >     a) do not use session parameter in public DB API methods.
> >     b) fix places where we are doing N queries in N transactions instead
> > of 1.
> >     c) get only data that is used (e.g. len(query.all()) =>
> query.count()).
> >
> > ----
> >
> > DB Migrations
> >
> >   *) Test DB Migrations against all backends and real data.
> >
> >   *) Fix: DB schemas after Migrations should be same in different
> backends
> >
> >   *) Fix: hidden bugs, that are caused by wrong migrations:
> >      a) fix indexes. e.g. 152 migration in Nova drop all Indexes that
> > has deleted column
> >      b) fix wrong types
> >      c) drop unused tables
> >
> >   *) Switch from sqlalchemy-migrate to something that is not death (e.g.
> > alembic).
> >
> > ----
> >
> > DB Models
> >
> >   *) Fix: Schema that is created by Models should be the same as after
> > migrations.
> >
> >   *) Fix: Unit tests should be runed on DB that was created by Models
> > not migrations.
> >
> >   *) Add test that checks that Models are synced with migrations.
> >
> > ----
> >
> > Oslo Code
> >
> >   *) Base Sqlalchemy Models.
> >
> >   *) Work around engine and session.
> >
> >   *) SqlAlchemy Utils - that helps us with migrations and tests.
> >
> >   *) Test migrations Base.
> >
> >   *) Use common test wrapper that allows us to run tests on different
> > backends.
> >
> >
> >
> ###############################################################################
> >                                                        Implementation
> >
> ###############################################################################
> >
> >   This is really really huge task. And we are almost done with Nova=).
> >
> >   In OpenStack for such work there is only one approach (“baby steps”
> > development deriven). So we are making tons of patches that could be
> > easy reviewed. But there is also minuses in such approach. It is pretty
> > hard to track work on high level. And sometimes there are misunderstand.
> >
> >   For example with oslo code. In few words at this moment we would like
> > to add (for some time) in oslo monkey patching for sqlalchemy-migrate.
> > And I got reasonable question from Doug Hellmann. Why? I answer because
> > of our “baby steps”. But if you don’t have a list of baby steps it is
> > pretty hard to understand why our baby steps need this thing. And why we
> > don’t switch to alembic firstly. So I would like to describe our Road
> > Map and write list of "baby steps".
> >
> >
> > -------
> >
> > OSLO
> >
> >   *) (Merged) Base code for Models and sqlalchemy engine (session)
> >
> >   *) (On review) Sqlalchemy utils that are used to:
> >       1. Fix bugs in sqlalchemy-migrate
> >       2. Base code for migrations that provides Unique Constraints.
> >       3. Utils for db.archiving helps us to create and check shadow
> tables.
> >
> >   *) (On review) Testtools wrapper
> >        We should have only one testtool wrapper in all projects. And
> > this is the one of base steps in task of running tests against all
> backends.
> >
> >   *) (On review) Test migrations base
> >        Base classes that provides us to test our migrations against all
> > backends on real data
> >
> >   *) (On review, not finished yet) DB Reconnect.
> >
> >   *) (Not finished) Test that checks that schemas and models are synced
> >
> > -------
> >
> > ${PROJECT_NAME}
> >
> >
> > In different projects we could work absolutely simultaneously, and first
> > candidates are Glance and Cinder. But inside project we could also work
> > simultaneously. Here is the workflow:
> >
> >
> >   1) (SYNC) Use base code for Models and sqlalchemy engines (from oslo)
> >
> >   2) (SYNC) Use test migrations base (from oslo)
> >
> >   3) (SYNC) Use SqlAlchemy utils (from oslo)
> >
> >   4) (1 patch) Switch to OSLO DB code
> >
> >   5) (1 patch) Remove ported test migrations
> >
> >   6) (1 Migration) Provide unique constraints (change type of “deleted”
> > column)
> >
> >   7) (1 Migration) Add shadow tables
> >     a) Create shadow tables
> >     b) Add test that checks that they are synced always
> >
> >   8) (N Migrations) UniqueConstraint/Session/Optimization workflow:
> >     a) (1 patch) Add/Improve/Refactor tests for part of api (that is
> > connected with model)
> >     b) (1 patch) Fix session
> >     c) (1 patch)  Optimize method
> >     d) if required (1 Migration) Add missing Unique Constraints.
> >
> >   9) (M Migrations) Sync Models with DB schemas (almost independent from
> 8)
> >     a) (K pathces) Sync Models with MySQL Migrations
> >         (because they are different in different backneds)
> >     b) (Fix, L migrations) Fix migrations, that have bugs (e.g. 152
> > migraiton in Nova, drops all
> >          Indexes that have ‘deleted’ column).
> >     c) Add test for MySql
> >     d) (Fix, 2 migrations) Sync other backends schemas with MySql schema
> >     e) Add test for SQLite and Psql
> >     f) Rethink our models:
> >        *) Improve Indexes
> >        *) Use Enum where it is required
> >        *) Fix bugs if there are.
> >        *) Add missing default values in schema.
> >
> > 10) Test should be runed on schema that is created from Models, not
> > migrations (depend on 9)
> >
> > 11) Use alembic (or something else) instead of sqlalchemy-migrate (after
> > 9.e. it will be safe
> >   a) make test migrations independent from sqlalchemy-migrate
> >   b) (If alembic) Provide work with sqlite.
> >   c) Replace all migrations
> >
> > Main part of work is in 7, 8 and 10 points.
> >
> >
> >
> ###############################################################################
> >                                            Answer for Doug's question
> >
> ###############################################################################
> >
> > Question:
> >   Why we should put in oslo slqlalchemy-migrate monkey patches, when we
> > are planing to switch to alembic?
> >
> > Answer:
> >    If we don’t put in oslo sqlalchemy-migrate monkey patches. We won't
> > be able to work on 7 point at all until 8 and 10 points will be
> > implemented in every project. Also work around 8 point is not finished,
> > so we are not able to implement 10 points in any of project. So this
> > blocks almost all work in all projects. I think that these 100-200 lines
> > of code are not so big price for saving few cycles of time.
> >
> >
> >
> >
> ###############################################################################
> >                                            Current progress
> >
> ###############################################################################
> >
> >
> > Nova
> >
> >   *) 1,4,6,7 - fully finished
> >
> >   *) 2,3,5 - waiting for oslo, but pretty simple to sync it
> >
> >   *) 8 - almost finished
> >     a) Few patched around tests (on review)
> >     b) 9 patches that add UniqueConstraints (on review) and few missing
> >     c) Few patches about session (on review) and few missing
> >
> >   *) 9 - tons of patches that syncs models with MySql on review. Also we
> > are working around tests that checks that schemas that are crated by
> > models and migrations are same. So it will be a lot of work here.
> >
> >   *) 10 - pretty simple (blocked by 9)
> >
> >   *) 11 - tons of work, candidate for I cycle.
> >
> >   *) Also here is work around run test on all backends but it is not
> > finished yet.
> >
> > -------
> >
> > Cinder
> >
> >    *) 1,4 - finished
> >
> >    *) Some work around 8 (sessions and tests)
> >
> >    *) Other things are blocked by oslo
> >
> > -------
> >
> > Glance
> >
> >    *) 1 - finished
> >
> >    *) 4 - working on it
> >
> >
> >
> > Best regards,
> > Boris Pavlović
> >
> > (irc boris-42)
> >
> >
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130703/e8283bdd/attachment.html>