Open Stack

On 08/16/2013 09:31 AM, Victor Sergeyev wrote:
> Hello All.
> 
> Glance cores (Mark Washenberger, Flavio Percoco, Iccha Sethi) have some
> questions about Oslo DB code, and why is it so important to use it
> instead of custom implementation and so on. As there were a lot of
> questions it was really hard to answer on all this questions in IRC. So
> we decided that mailing list is better place for such things.

There is another main point - which is at the last summit, we talked
about various legit database things that need to be done to support CD
and rolling deploys. The list is not small, and it's a task that's
important. Needing to implement it in all of the projects separately is
kind of an issue, whereas if the projects are all using the database the
same way, then the database team can engineer the same mechanisms for
doing rolling schema changes, and then operators can have a consistent
expectation when they're running a cloud.

> List of main questions:
> 
> 1. What includes oslo DB code?  
> 2. Why is it safe to replace custom implementation by Oslo DB code? 
> 3. Why oslo DB code is better than custom implementation?
> 4. Why oslo DB code won’t slow up project development progress?
> 5. What we are going actually to do in Glance?
> 6. What is the current status?
> 
> Answers:
> 
> 1. What includes oslo DB code?
> 
> Currently Oslo code improves different aspects around DB:
> -- Work with SQLAlchemy models, engine and session
> -- Lot of tools for work with SQLAlchemy 
> -- Work with unique keys
> -- Base test case for work with database
> -- Test migrations against different backends
> -- Sync DB Models with actual schemas in DB (add test that they are
> equivalent)
> 
> 
> 2. Why is it safe to replace custom implementation by Oslo DB code? 
> 
> Oslo module, as base openstack module, takes care about code quality.
> Usually, common code more readable (most of flake8 checks enabled in
> Oslo) and have better test coverage.  Also it was tested in different
> use-cases (in production also) in an other projects so bugs in Oslo code
> were already fixed. So we can be sure, that we use high-quality code.
> 
> 
> 3. Why oslo DB code is better than custom implementation?
> 
> There are some arguments pro Oslo database code 
> 
> -- common code collects useful features from different projects
> Different utils, for work with database, common test class, module for
> database migration, and  other features are already in Oslo db code.
> Patch on automatic retry db.api query if db connection lost on review at
> the moment. If we use Oslo db code we should not care, how to port these
> (and others - in the future) features to Glance - it will came to all
> projects automaticly when it will came to Oslo. 
> 
> -- unified project work with database
> As it was already said,  It can help developers work with database in a
> same way in different projects. It’s useful if developer work with db in
> a few projects - he use same base things and got no surprises from them. 
> 
> -- it’s will reduce time for running tests.
> Maybe it’s minor feature, but it’s also can be important. We can removed
> some tests for base `DB` classes (such as session, engines, etc)  and
> replaced for work with DB to mock calls.
> 
> 
> 4. Why oslo DB code won’t slow up project development progress?
> 
> Oslo code for work with database already in such projects as Nova,
> Neutron, Celiometer and Ironic. AFAIK, these projects development speed
> doesn’t decelerated (please fix me, If I’m wrong). Work with database
> level already improved and tested in Oslo project, so we can concentrate
> on work with project features. All features, that already came to oslo
> code will be available in Glance, but if you want to add some specific
> feature to project *just now* you will be able to do it in project code.
> 
> 
> 5. What we are going actually to do in Glance?
> 
> -- Improve test coverage of DB API layer
> We are going to increase test coverage of glance/db/sqlalchemy/api
> module and fix bugs, if found. 
> 
> -- Run DB API tests on all backends
> -- Use Oslo migrations base test case for test migrations against
> different backends
> There are lot of different things in SQl backends. For example work with
> casting.
> In current SQLite we are able to store everything in column (with any
> type). Mysql will try to convert value to required type, and postgresql
> will raise IntegrityError. 
> If we will improve this feature, we will be sure, that all Glance DB
> migrations will run correctly on all backends.
> 
> -- Use Oslo code for SA models, engine and session
> -- Use Oslo SA utils
> Using common code for work with database was already discussed and
> approved for all projects. So we are going to implement common code for
> work with database instead of Glance implementation.
> 
> -- Fix work with session and transactions
> Our work items in Glance:
> - don't pass session instances to public DB methods
> - use explicit transactions only when necessary
> - fix incorrect usage of sessions throughout the DB-related code
> 
> -- Optimize methods
> When we will have tests for all functions in glance/db/sqlalchemy/api
> module it’s will be safe to refactor api methods. It will make these
> functions more clean, readable and faster.
> 
> The main ideas are:
> - identify and remove unused methods
> - consolidate duplicate methods when possible
> - ensure SQLAlchemy objects are not leaking out of the API
> - ensure related methods are grouped together and named consistently
> 
> -- Add missing unique constraints
> We should provide missed unique constraints, based on database queries
> from glance.db.sqlalchemy.api module. It’s will reduce data duplication
> and became one more step to Glance database normalization.
> 
> -- Sync models definitions with DB migrate scripts
> At the moment we do not use Glance db models for db creation. To create
> db we use migrations. We don't have any tests that checks, that our
> models are up-to-date. Also we are testing it only on sqlite, which
> couldn't test such things as nullable constraints.
> So we plain fix all mistakes in models and migration, sync effects of
> migrations in different backends and add tests that ensures that models
> are up-to-date.
> -- Use alembic for database migrations 
> SQLAlchemy-migrate (Glance database migration tool) is fine for
> straight-line migrations, but does not really support branching, needed
> for clean backports of some but not all migrations to a stable branch.
> Also this project doesn't seems to be live.
> Alembic has better branching support and will meet our needs better. We
> should switch migration frameworks, and convert our existing migration
> scripts to Alembic syntax.
> 
> -- DB Reconnect
> There are a variety of circumstances which can cause a transient failure
> in database connections, for example: restart / upgrade of the database,
> just a network failure, etc. Glance (as all projects connecting to a
> database) would benefit from the db/api catching these
> "db-has-gone-away" errors and automatically reconnecting and retrying
> the last db api method call. It is not necessary to abort long-running
> operations (such as glance image-create) just because of a momentary
> interruption in db connectivity.
> 
> -- No downtime database upgrade
> No comments
> 
> 
> 6. What is the current status?
> 
> -- Improve test coverage of DB API layer
> Not started
> 
> -- Run DB API tests on all backends
> Not started
> 
> -- Use Oslo code for SA models, engine and session
> Started
> 
> -- Use Oslo migrations base test case for test migrations against
> different backends
> -- Use Oslo SA utils
> In progress at the moment. During the implementation, such changes was made:
> - DB layer code cleanup. We removed project-specific functions for work
> with DB layer in db.api. Now we use common Oslo code.
> - Base class for DB models from common code was used, instead of project
> base model class.
> - Refactored migration module. Also we plan to use common module for
> migrations from Oslo 
> - Race condition in migration 012 was fixed.
> - Tests, based on module glance.tests.integration.legacy_functional.base
> use sqlite in memory now.
> - Modified work with config - we use some config values from Oslo code.
> - Removed some tests for base class for work with DB - functions for
> work with DB was already tested in Oslo project.
> - Renamed unique constraints due to common name convention. 
> - Patched sqlalchemy-migrate to fix UC bugs in SQLite.
> 
> -- Fix work with session and transactions
> Not started
> 
> -- Optimize methods
> Not started
> 
> -- Add missing unique constraints
> Not started
> 
> -- Sync models definitions with DB migrate scripts
> Mostly done. 
> 
> -- Use alembic for database migrations 
> Not started
> 
> -- DB Reconnect
> On review in Oslo. We will be able to use this feature after we will fix
> work with session and transactions
> 
> Thanks, Victor.
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>