Open Stack

On 08/27/2013 05:32 AM, Boris Pavlovic wrote:
> Jay, 
> 
> I should probably share to you about our work around DB.
> 
> Migrations should be run only in production and only for production
> backends (e.g. psql and mysql) 
> In tests we should use Schemas created by Models
> (BASE.metadata.create_all())
> 
> We are not able to use in this approach in moment  because we don't have
> any mechanism to check that MODELS and SCHEMAS are EQUAL. 
> And actually MODELS and SCHEMAS are DIFFERENT. 
> 
> E.g. in Celiometer we have BP that syncs models and migration
> https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-db-sync-models-with-migrations
> (in other projects we are doing the same)
> 
> And also we are working around (oslo) generic tests that checks that
> models and migrations are equal: 
> https://review.openstack.org/#/c/42307/
> 
> 
> So in our roadmap (in this case is):
> 1) Soft switch to alembic (with code that allows to have sqla-migrate
> and alembic migration in the same time)
> 2) Sync Models and Migrations (fix DB schemas also)
> 3) Add from oslo generic test that checks all this stuff
> 4) Use BASE.create_all() for Schema creation instead of migrations. 
> 
> 
> But in OpenStack is not so simple to implement such huge changes, so it
> take some time=)

Hmm, so I'm not sure how to proceed here?

Should we be using alembic or the older migrations right now?

Are there any examples of what you're proposing here?


> 
> 
> Best regards,
> Boris Pavlovic
> ---
> Mirantis Inc. 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On Tue, Aug 27, 2013 at 12:02 AM, Jay Pipes <jaypipes at gmail.com
> <mailto:jaypipes at gmail.com>> wrote:
> 
>     On 08/26/2013 03:40 PM, Herndon, John Luke (HPCS - Ft. Collins) wrote:
> 
>         Jay -
> 
>         It looks there is an error in the migration script that causes
>         it to abort:
> 
>         AttributeError: 'ForeignKeyConstraint' object has no attribute
>         'drop'
> 
>         My guess is the migration runs on the first test, creates event
>         types
>         table fine, but exits with the above error, so migration is not
>         "complete". Thus every subsequent test tries to migrate the db, and
>         notices that event types already exists.
> 
> 
>     I'd corrected that particular mistake and pushed an updated
>     migration script.
> 
>     Best,
>     -jay
> 
> 
> 
>         -john
> 
>         On 8/26/13 1:15 PM, "Jay Pipes" <jaypipes at gmail.com
>         <mailto:jaypipes at gmail.com>> wrote:
> 
>             I just noticed that every single test case for SQL-driver
>             storage is
>             executing every single migration upgrade before every single
>             test case
>             run:
> 
>             https://github.com/openstack/__ceilometer/blob/master/__ceilometer/tests/db.py
>             <https://github.com/openstack/ceilometer/blob/master/ceilometer/tests/db.py>
>             #L46
> 
>             https://github.com/openstack/__ceilometer/blob/master/__ceilometer/storage/imp
>             <https://github.com/openstack/ceilometer/blob/master/ceilometer/storage/imp>
>             l_sqlalchemy.py#L153
> 
>             instead of simply creating a new database schema from the
>             models in the
>             current source code base using a call to
>             sqlalchemy.MetaData.create___all().
> 
>             This results in re-running migrations over and over again,
>             instead of
>             having dedicated migration tests that would test each migration
>             individually, as is the case in projects like Glance...
> 
>             Is this intentional?
> 
>             Best,
>             -jay
> 
>             On 08/26/2013 02:59 PM, Sandy Walsh wrote:
> 
>                 I'm getting the same problem with a different migration
>                 (mine is
>                 complaining that a column already exists)
> 
>                 http://paste.openstack.org/__show/44512/
>                 <http://paste.openstack.org/show/44512/>
> 
>                 I've compared it to the other migrations and it seems fine.
> 
>                 -S
> 
>                 On 08/26/2013 02:34 PM, Jay Pipes wrote:
> 
>                     Hey all,
> 
>                     I'm trying to figure out what is going wrong with my
>                     code for this
>                     patch:
> 
>                     https://review.openstack.org/__41316
>                     <https://review.openstack.org/41316>
> 
>                     I had previously added a sqlalchemy-migrate
>                     migration script to add an
>                     event_type table, and had that working, but then was
>                     asked to instead
>                     use Alembic for migrations. So, I removed the
>                     sqlalchemy-migrate
>                     migration file and added an Alembic migration [1].
> 
>                     Unfortunately, I am getting the following error when
>                     running tests:
> 
>                     OperationalError: (OperationalError) table
>                     event_type already exists
>                     u'\nCREATE TABLE event_type (\n\tid INTEGER NOT
>                     NULL, \n\t"desc"
>                     VARCHAR(255), \n\tPRIMARY KEY (id), \n\tUNIQUE
>                     ("desc")\n)\n\n' ()
> 
>                     The migration adds the event_type table. I've seen
>                     this error occur
>                     before when using SQLite due to SQLite's ALTER TABLE
>                     statement not
>                     allowing the rename of a column. In the
>                     sqlalchemy-migrate migration, I
>                     had a specialized SQLite migration upgrade [2] and
>                     downgrade [3]
>                     script,
>                     but I'm not sure how I am supposed to handle this in
>                     Alembic. Could
>                     someone help me out?
> 
>                     Thanks,
>                     -jay
> 
>                     [1]
> 
>                     https://review.openstack.org/#__/c/41316/16/ceilometer/__storage/sqlalchemy/
>                     <https://review.openstack.org/#/c/41316/16/ceilometer/storage/sqlalchemy/>
>                     alembic/versions/49036daaaafd___add_event_types.py
> 
>                     [2]
> 
>                     https://review.openstack.org/#__/c/41316/14/ceilometer/__storage/sqlalchemy/
>                     <https://review.openstack.org/#/c/41316/14/ceilometer/storage/sqlalchemy/>
>                     migrate_repo/versions/013___sqlite_upgrade.sql
> 
>                     [3]
> 
>                     https://review.openstack.org/#__/c/41316/14/ceilometer/__storage/sqlalchemy/
>                     <https://review.openstack.org/#/c/41316/14/ceilometer/storage/sqlalchemy/>
>                     migrate_repo/versions/013___sqlite_downgrade.sql
> 
> 
>                     _________________________________________________
>                     OpenStack-dev mailing list
>                     OpenStack-dev at lists.openstack.__org
>                     <mailto:OpenStack-dev at lists.openstack.org>
>                     http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>                     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
> 
> 
> 
>             _________________________________________________
>             OpenStack-dev mailing list
>             OpenStack-dev at lists.openstack.__org
>             <mailto:OpenStack-dev at lists.openstack.org>
>             http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>             <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
> 
> 
>             _________________________________________________
>             OpenStack-dev mailing list
>             OpenStack-dev at lists.openstack.__org
>             <mailto:OpenStack-dev at lists.openstack.org>
>             http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>             <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
> 
> 
> 
>     _________________________________________________
>     OpenStack-dev mailing list
>     OpenStack-dev at lists.openstack.__org
>     <mailto:OpenStack-dev at lists.openstack.org>
>     http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
> 
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>