Open Stack

###############################################################################
                                                                Goal
###############################################################################

We should fix work with DB, unify it in all projects and use oslo code for
all common things.

In more words:

DB API

  *) Fully cover by tests.

  *) Run tests against all backends (now they are runed only against
sqlite).

  *) Unique constraints (instead of select + insert)
     a) Provide unique constraints.
     b) Add missing unique constraints.

  *) DB Archiving
     a) create shadow tables
     b) add tests that checks that shadow and main table are synced.
     c) add code that work with shadow tables.

  *) DB API performance optimization
    a) Remove unused joins.
    b) 1 query instead of N (where it is possible).
    c) Add methods that could improve performance.
    d) Drop unused methods.

  *) DB reconnect
    a) Don't break huge task if we lost connection for a moment. just retry
DB query.

  *) DB Session cleanup
    a) do not use session parameter in public DB API methods.
    b) fix places where we are doing N queries in N transactions instead of
1.
    c) get only data that is used (e.g. len(query.all()) => query.count()).

----

DB Migrations

  *) Test DB Migrations against all backends and real data.

  *) Fix: DB schemas after Migrations should be same in different backends

  *) Fix: hidden bugs, that are caused by wrong migrations:
     a) fix indexes. e.g. 152 migration in Nova drop all Indexes that has
deleted column
     b) fix wrong types
     c) drop unused tables

  *) Switch from sqlalchemy-migrate to something that is not death (e.g.
alembic).

----

DB Models

  *) Fix: Schema that is created by Models should be the same as after
migrations.

  *) Fix: Unit tests should be runed on DB that was created by Models not
migrations.

  *) Add test that checks that Models are synced with migrations.

----

Oslo Code

  *) Base Sqlalchemy Models.

  *) Work around engine and session.

  *) SqlAlchemy Utils - that helps us with migrations and tests.

  *) Test migrations Base.

  *) Use common test wrapper that allows us to run tests on different
backends.


###############################################################################
                                                       Implementation
###############################################################################

  This is really really huge task. And we are almost done with Nova=).

  In OpenStack for such work there is only one approach ("baby steps"
development deriven). So we are making tons of patches that could be easy
reviewed. But there is also minuses in such approach. It is pretty hard to
track work on high level. And sometimes there are misunderstand.

  For example with oslo code. In few words at this moment we would like to
add (for some time) in oslo monkey patching for sqlalchemy-migrate. And I
got reasonable question from Doug Hellmann. Why? I answer because of our
"baby steps". But if you don't have a list of baby steps it is pretty hard
to understand why our baby steps need this thing. And why we don't switch
to alembic firstly. So I would like to describe our Road Map and write list
of "baby steps".


-------

OSLO

  *) (Merged) Base code for Models and sqlalchemy engine (session)

  *) (On review) Sqlalchemy utils that are used to:
      1. Fix bugs in sqlalchemy-migrate
      2. Base code for migrations that provides Unique Constraints.
      3. Utils for db.archiving helps us to create and check shadow tables.

  *) (On review) Testtools wrapper
       We should have only one testtool wrapper in all projects. And this
is the one of base steps in task of running tests against all backends.

  *) (On review) Test migrations base
       Base classes that provides us to test our migrations against all
backends on real data

  *) (On review, not finished yet) DB Reconnect.

  *) (Not finished) Test that checks that schemas and models are synced

-------

${PROJECT_NAME}


In different projects we could work absolutely simultaneously, and first
candidates are Glance and Cinder. But inside project we could also work
simultaneously. Here is the workflow:


  1) (SYNC) Use base code for Models and sqlalchemy engines (from oslo)

  2) (SYNC) Use test migrations base (from oslo)

  3) (SYNC) Use SqlAlchemy utils (from oslo)

  4) (1 patch) Switch to OSLO DB code

  5) (1 patch) Remove ported test migrations

  6) (1 Migration) Provide unique constraints (change type of "deleted"
column)

  7) (1 Migration) Add shadow tables
    a) Create shadow tables
    b) Add test that checks that they are synced always

  8) (N Migrations) UniqueConstraint/Session/Optimization workflow:
    a) (1 patch) Add/Improve/Refactor tests for part of api (that is
connected with model)
    b) (1 patch) Fix session
    c) (1 patch)  Optimize method
    d) if required (1 Migration) Add missing Unique Constraints.

  9) (M Migrations) Sync Models with DB schemas (almost independent from 8)
    a) (K pathces) Sync Models with MySQL Migrations
        (because they are different in different backneds)
    b) (Fix, L migrations) Fix migrations, that have bugs (e.g. 152
migraiton in Nova, drops all
         Indexes that have 'deleted' column).
    c) Add test for MySql
    d) (Fix, 2 migrations) Sync other backends schemas with MySql schema
    e) Add test for SQLite and Psql
    f) Rethink our models:
       *) Improve Indexes
       *) Use Enum where it is required
       *) Fix bugs if there are.
       *) Add missing default values in schema.

10) Test should be runed on schema that is created from Models, not
migrations (depend on 9)

11) Use alembic (or something else) instead of sqlalchemy-migrate (after
9.e. it will be safe
  a) make test migrations independent from sqlalchemy-migrate
  b) (If alembic) Provide work with sqlite.
  c) Replace all migrations

Main part of work is in 7, 8 and 10 points.


###############################################################################
                                           Answer for Doug's question
###############################################################################

Question:
  Why we should put in oslo slqlalchemy-migrate monkey patches, when we are
planing to switch to alembic?

Answer:
   If we don't put in oslo sqlalchemy-migrate monkey patches. We won't be
able to work on 7 point at all until 8 and 10 points will be implemented in
every project. Also work around 8 point is not finished, so we are not able
to implement 10 points in any of project. So this blocks almost all work in
all projects. I think that these 100-200 lines of code are not so big price
for saving few cycles of time.



###############################################################################
                                           Current progress
###############################################################################


Nova

  *) 1,4,6,7 - fully finished

  *) 2,3,5 - waiting for oslo, but pretty simple to sync it

  *) 8 - almost finished
    a) Few patched around tests (on review)
    b) 9 patches that add UniqueConstraints (on review) and few missing
    c) Few patches about session (on review) and few missing

  *) 9 - tons of patches that syncs models with MySql on review. Also we
are working around tests that checks that schemas that are crated by models
and migrations are same. So it will be a lot of work here.

  *) 10 - pretty simple (blocked by 9)

  *) 11 - tons of work, candidate for I cycle.

  *) Also here is work around run test on all backends but it is not
finished yet.

-------

Cinder

   *) 1,4 - finished

   *) Some work around 8 (sessions and tests)

   *) Other things are blocked by oslo

-------

Glance

   *) 1 - finished

   *) 4 - working on it



Best regards,
Boris Pavlović

(irc boris-42)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130702/24ed6d78/attachment.html>