Open Stack

Jay,

I should probably share to you about our work around DB.

Migrations should be run only in production and only for production
backends (e.g. psql and mysql)
In tests we should use Schemas created by Models
(BASE.metadata.create_all())

We are not able to use in this approach in moment  because we don't have
any mechanism to check that MODELS and SCHEMAS are EQUAL.
And actually MODELS and SCHEMAS are DIFFERENT.

E.g. in Celiometer we have BP that syncs models and migration
https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-db-sync-models-with-migrations
(in other projects we are doing the same)

And also we are working around (oslo) generic tests that checks that models
and migrations are equal:
https://review.openstack.org/#/c/42307/


So in our roadmap (in this case is):
1) Soft switch to alembic (with code that allows to have sqla-migrate and
alembic migration in the same time)
2) Sync Models and Migrations (fix DB schemas also)
3) Add from oslo generic test that checks all this stuff
4) Use BASE.create_all() for Schema creation instead of migrations.


But in OpenStack is not so simple to implement such huge changes, so it
take some time=)


Best regards,
Boris Pavlovic
---
Mirantis Inc.










On Tue, Aug 27, 2013 at 12:02 AM, Jay Pipes <jaypipes at gmail.com> wrote:

> On 08/26/2013 03:40 PM, Herndon, John Luke (HPCS - Ft. Collins) wrote:
>
>> Jay -
>>
>> It looks there is an error in the migration script that causes it to
>> abort:
>>
>> AttributeError: 'ForeignKeyConstraint' object has no attribute 'drop'
>>
>> My guess is the migration runs on the first test, creates event types
>> table fine, but exits with the above error, so migration is not
>> "complete". Thus every subsequent test tries to migrate the db, and
>> notices that event types already exists.
>>
>
> I'd corrected that particular mistake and pushed an updated migration
> script.
>
> Best,
> -jay
>
>
>
>  -john
>>
>> On 8/26/13 1:15 PM, "Jay Pipes" <jaypipes at gmail.com> wrote:
>>
>>  I just noticed that every single test case for SQL-driver storage is
>>> executing every single migration upgrade before every single test case
>>> run:
>>>
>>> https://github.com/openstack/**ceilometer/blob/master/**
>>> ceilometer/tests/db.py<https://github.com/openstack/ceilometer/blob/master/ceilometer/tests/db.py>
>>> #L46
>>>
>>> https://github.com/openstack/**ceilometer/blob/master/**
>>> ceilometer/storage/imp<https://github.com/openstack/ceilometer/blob/master/ceilometer/storage/imp>
>>> l_sqlalchemy.py#L153
>>>
>>> instead of simply creating a new database schema from the models in the
>>> current source code base using a call to sqlalchemy.MetaData.create_**
>>> all().
>>>
>>> This results in re-running migrations over and over again, instead of
>>> having dedicated migration tests that would test each migration
>>> individually, as is the case in projects like Glance...
>>>
>>> Is this intentional?
>>>
>>> Best,
>>> -jay
>>>
>>> On 08/26/2013 02:59 PM, Sandy Walsh wrote:
>>>
>>>> I'm getting the same problem with a different migration (mine is
>>>> complaining that a column already exists)
>>>>
>>>> http://paste.openstack.org/**show/44512/<http://paste.openstack.org/show/44512/>
>>>>
>>>> I've compared it to the other migrations and it seems fine.
>>>>
>>>> -S
>>>>
>>>> On 08/26/2013 02:34 PM, Jay Pipes wrote:
>>>>
>>>>> Hey all,
>>>>>
>>>>> I'm trying to figure out what is going wrong with my code for this
>>>>> patch:
>>>>>
>>>>> https://review.openstack.org/**41316<https://review.openstack.org/41316>
>>>>>
>>>>> I had previously added a sqlalchemy-migrate migration script to add an
>>>>> event_type table, and had that working, but then was asked to instead
>>>>> use Alembic for migrations. So, I removed the sqlalchemy-migrate
>>>>> migration file and added an Alembic migration [1].
>>>>>
>>>>> Unfortunately, I am getting the following error when running tests:
>>>>>
>>>>> OperationalError: (OperationalError) table event_type already exists
>>>>> u'\nCREATE TABLE event_type (\n\tid INTEGER NOT NULL, \n\t"desc"
>>>>> VARCHAR(255), \n\tPRIMARY KEY (id), \n\tUNIQUE ("desc")\n)\n\n' ()
>>>>>
>>>>> The migration adds the event_type table. I've seen this error occur
>>>>> before when using SQLite due to SQLite's ALTER TABLE statement not
>>>>> allowing the rename of a column. In the sqlalchemy-migrate migration, I
>>>>> had a specialized SQLite migration upgrade [2] and downgrade [3]
>>>>> script,
>>>>> but I'm not sure how I am supposed to handle this in Alembic. Could
>>>>> someone help me out?
>>>>>
>>>>> Thanks,
>>>>> -jay
>>>>>
>>>>> [1]
>>>>>
>>>>> https://review.openstack.org/#**/c/41316/16/ceilometer/**
>>>>> storage/sqlalchemy/<https://review.openstack.org/#/c/41316/16/ceilometer/storage/sqlalchemy/>
>>>>> alembic/versions/49036daaaafd_**add_event_types.py
>>>>>
>>>>> [2]
>>>>>
>>>>> https://review.openstack.org/#**/c/41316/14/ceilometer/**
>>>>> storage/sqlalchemy/<https://review.openstack.org/#/c/41316/14/ceilometer/storage/sqlalchemy/>
>>>>> migrate_repo/versions/013_**sqlite_upgrade.sql
>>>>>
>>>>> [3]
>>>>>
>>>>> https://review.openstack.org/#**/c/41316/14/ceilometer/**
>>>>> storage/sqlalchemy/<https://review.openstack.org/#/c/41316/14/ceilometer/storage/sqlalchemy/>
>>>>> migrate_repo/versions/013_**sqlite_downgrade.sql
>>>>>
>>>>>
>>>>> ______________________________**_________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.**org<OpenStack-dev at lists.openstack.org>
>>>>> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>>>>
>>>>
>>>
>>> ______________________________**_________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.**org <OpenStack-dev at lists.openstack.org>
>>> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>>
>>>
>>> ______________________________**_________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.**org <OpenStack-dev at lists.openstack.org>
>>> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>>
>>
>
> ______________________________**_________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.**org <OpenStack-dev at lists.openstack.org>
> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130827/11bc2e2d/attachment.html>