Open Stack

Hello All.

Glance cores (Mark Washenberger, Flavio Percoco, Iccha Sethi) have some
questions about Oslo DB code, and why is it so important to use it instead
of custom implementation and so on. As there were a lot of questions it was
really hard to answer on all this questions in IRC. So we decided that
mailing list is better place for such things.

List of main questions:

1. What includes oslo DB code?
2. Why is it safe to replace custom implementation by Oslo DB code?
3. Why oslo DB code is better than custom implementation?
4. Why oslo DB code won’t slow up project development progress?
5. What we are going actually to do in Glance?
6. What is the current status?

Answers:

1. What includes oslo DB code?

Currently Oslo code improves different aspects around DB:
-- Work with SQLAlchemy models, engine and session
-- Lot of tools for work with SQLAlchemy
-- Work with unique keys
-- Base test case for work with database
-- Test migrations against different backends
-- Sync DB Models with actual schemas in DB (add test that they are
equivalent)


2. Why is it safe to replace custom implementation by Oslo DB code?

Oslo module, as base openstack module, takes care about code quality.
Usually, common code more readable (most of flake8 checks enabled in Oslo)
and have better test coverage.  Also it was tested in different use-cases
(in production also) in an other projects so bugs in Oslo code were already
fixed. So we can be sure, that we use high-quality code.


3. Why oslo DB code is better than custom implementation?

There are some arguments pro Oslo database code

-- common code collects useful features from different projects
Different utils, for work with database, common test class, module for
database migration, and  other features are already in Oslo db code. Patch
on automatic retry db.api query if db connection lost on review at the
moment. If we use Oslo db code we should not care, how to port these (and
others - in the future) features to Glance - it will came to all projects
automaticly when it will came to Oslo.

-- unified project work with database
As it was already said,  It can help developers work with database in a
same way in different projects. It’s useful if developer work with db in a
few projects - he use same base things and got no surprises from them.

-- it’s will reduce time for running tests.
Maybe it’s minor feature, but it’s also can be important. We can removed
some tests for base `DB` classes (such as session, engines, etc)  and
replaced for work with DB to mock calls.


4. Why oslo DB code won’t slow up project development progress?

Oslo code for work with database already in such projects as Nova, Neutron,
Celiometer and Ironic. AFAIK, these projects development speed doesn’t
decelerated (please fix me, If I’m wrong). Work with database level already
improved and tested in Oslo project, so we can concentrate on work with
project features. All features, that already came to oslo code will be
available in Glance, but if you want to add some specific feature to
project *just now* you will be able to do it in project code.


5. What we are going actually to do in Glance?

-- Improve test coverage of DB API layer
We are going to increase test coverage of glance/db/sqlalchemy/api module
and fix bugs, if found.

-- Run DB API tests on all backends
-- Use Oslo migrations base test case for test migrations against different
backends
There are lot of different things in SQl backends. For example work with
casting.
In current SQLite we are able to store everything in column (with any
type). Mysql will try to convert value to required type, and postgresql
will raise IntegrityError.
If we will improve this feature, we will be sure, that all Glance DB
migrations will run correctly on all backends.

-- Use Oslo code for SA models, engine and session
-- Use Oslo SA utils
Using common code for work with database was already discussed and approved
for all projects. So we are going to implement common code for work with
database instead of Glance implementation.

-- Fix work with session and transactions
Our work items in Glance:
- don't pass session instances to public DB methods
- use explicit transactions only when necessary
- fix incorrect usage of sessions throughout the DB-related code

-- Optimize methods
When we will have tests for all functions in glance/db/sqlalchemy/api
module it’s will be safe to refactor api methods. It will make these
functions more clean, readable and faster.

The main ideas are:
- identify and remove unused methods
- consolidate duplicate methods when possible
- ensure SQLAlchemy objects are not leaking out of the API
- ensure related methods are grouped together and named consistently

-- Add missing unique constraints
We should provide missed unique constraints, based on database queries from
glance.db.sqlalchemy.api module. It’s will reduce data duplication and
became one more step to Glance database normalization.

-- Sync models definitions with DB migrate scripts
At the moment we do not use Glance db models for db creation. To create db
we use migrations. We don't have any tests that checks, that our models are
up-to-date. Also we are testing it only on sqlite, which couldn't test such
things as nullable constraints.
So we plain fix all mistakes in models and migration, sync effects of
migrations in different backends and add tests that ensures that models are
up-to-date.
-- Use alembic for database migrations
SQLAlchemy-migrate (Glance database migration tool) is fine for
straight-line migrations, but does not really support branching, needed for
clean backports of some but not all migrations to a stable branch. Also
this project doesn't seems to be live.
Alembic has better branching support and will meet our needs better. We
should switch migration frameworks, and convert our existing migration
scripts to Alembic syntax.

-- DB Reconnect
There are a variety of circumstances which can cause a transient failure in
database connections, for example: restart / upgrade of the database, just
a network failure, etc. Glance (as all projects connecting to a database)
would benefit from the db/api catching these "db-has-gone-away" errors and
automatically reconnecting and retrying the last db api method call. It is
not necessary to abort long-running operations (such as glance
image-create) just because of a momentary interruption in db connectivity.

-- No downtime database upgrade
No comments


6. What is the current status?

-- Improve test coverage of DB API layer
Not started

-- Run DB API tests on all backends
Not started

-- Use Oslo code for SA models, engine and session
Started

-- Use Oslo migrations base test case for test migrations against different
backends
-- Use Oslo SA utils
In progress at the moment. During the implementation, such changes was made:
- DB layer code cleanup. We removed project-specific functions for work
with DB layer in db.api. Now we use common Oslo code.
- Base class for DB models from common code was used, instead of project
base model class.
- Refactored migration module. Also we plan to use common module for
migrations from Oslo
- Race condition in migration 012 was fixed.
- Tests, based on module glance.tests.integration.legacy_functional.base
use sqlite in memory now.
- Modified work with config - we use some config values from Oslo code.
- Removed some tests for base class for work with DB - functions for work
with DB was already tested in Oslo project.
- Renamed unique constraints due to common name convention.
- Patched sqlalchemy-migrate to fix UC bugs in SQLite.

-- Fix work with session and transactions
Not started

-- Optimize methods
Not started

-- Add missing unique constraints
Not started

-- Sync models definitions with DB migrate scripts
Mostly done.

-- Use alembic for database migrations
Not started

-- DB Reconnect
On review in Oslo. We will be able to use this feature after we will fix
work with session and transactions

Thanks, Victor.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130816/7e72c306/attachment.html>