[openstack-dev] [Ceilometer] Need help with Alembic...
Jay Pipes
jaypipes at gmail.com
Tue Aug 27 16:30:42 UTC 2013
On 08/27/2013 04:32 AM, Boris Pavlovic wrote:
> Jay,
>
> I should probably share to you about our work around DB.
>
> Migrations should be run only in production and only for production
> backends (e.g. psql and mysql)
> In tests we should use Schemas created by Models
> (BASE.metadata.create_all())
Agree on both.
> We are not able to use in this approach in moment because we don't have
> any mechanism to check that MODELS and SCHEMAS are EQUAL.
> And actually MODELS and SCHEMAS are DIFFERENT.
Sorry, I don't understand the connection... how does not having a
codified way of determining the difference between model and schema
(BTW, this does exist in sqlalchemy-migrate... look at the
compare_model_to_db method) not allow you to use metadata.create_all()
in tests or mean that you can't run migrations only in production?
> E.g. in Celiometer we have BP that syncs models and migration
> https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-db-sync-models-with-migrations
> (in other projects we are doing the same)
>
> And also we are working around (oslo) generic tests that checks that
> models and migrations are equal:
> https://review.openstack.org/#/c/42307/
OK, cool.
> So in our roadmap (in this case is):
> 1) Soft switch to alembic (with code that allows to have sqla-migrate
> and alembic migration in the same time)
I don't see the point in this at all... I would rather see patches that
just switch to Alembic and get rid of SQLAlchemy-migrate. Create an
initial Alembic migration that has the last state of the database schema
under SQLAlchemy-migrate... and then delete SA-Migrate.
> 2) Sync Models and Migrations (fix DB schemas also)
> 3) Add from oslo generic test that checks all this stuff
> 4) Use BASE.create_all() for Schema creation instead of migrations.
This is already done in some projects, IIRC... (Glance used to be this
way, at least)
> But in OpenStack is not so simple to implement such huge changes, so it
> take some time=)
>
>
> Best regards,
> Boris Pavlovic
> ---
> Mirantis Inc.
>
>
>
>
>
>
>
>
>
>
> On Tue, Aug 27, 2013 at 12:02 AM, Jay Pipes <jaypipes at gmail.com
> <mailto:jaypipes at gmail.com>> wrote:
>
> On 08/26/2013 03:40 PM, Herndon, John Luke (HPCS - Ft. Collins) wrote:
>
> Jay -
>
> It looks there is an error in the migration script that causes
> it to abort:
>
> AttributeError: 'ForeignKeyConstraint' object has no attribute
> 'drop'
>
> My guess is the migration runs on the first test, creates event
> types
> table fine, but exits with the above error, so migration is not
> "complete". Thus every subsequent test tries to migrate the db, and
> notices that event types already exists.
>
>
> I'd corrected that particular mistake and pushed an updated
> migration script.
>
> Best,
> -jay
>
>
>
> -john
>
> On 8/26/13 1:15 PM, "Jay Pipes" <jaypipes at gmail.com
> <mailto:jaypipes at gmail.com>> wrote:
>
> I just noticed that every single test case for SQL-driver
> storage is
> executing every single migration upgrade before every single
> test case
> run:
>
> https://github.com/openstack/__ceilometer/blob/master/__ceilometer/tests/db.py
> <https://github.com/openstack/ceilometer/blob/master/ceilometer/tests/db.py>
> #L46
>
> https://github.com/openstack/__ceilometer/blob/master/__ceilometer/storage/imp
> <https://github.com/openstack/ceilometer/blob/master/ceilometer/storage/imp>
> l_sqlalchemy.py#L153
>
> instead of simply creating a new database schema from the
> models in the
> current source code base using a call to
> sqlalchemy.MetaData.create___all().
>
> This results in re-running migrations over and over again,
> instead of
> having dedicated migration tests that would test each migration
> individually, as is the case in projects like Glance...
>
> Is this intentional?
>
> Best,
> -jay
>
> On 08/26/2013 02:59 PM, Sandy Walsh wrote:
>
> I'm getting the same problem with a different migration
> (mine is
> complaining that a column already exists)
>
> http://paste.openstack.org/__show/44512/
> <http://paste.openstack.org/show/44512/>
>
> I've compared it to the other migrations and it seems fine.
>
> -S
>
> On 08/26/2013 02:34 PM, Jay Pipes wrote:
>
> Hey all,
>
> I'm trying to figure out what is going wrong with my
> code for this
> patch:
>
> https://review.openstack.org/__41316
> <https://review.openstack.org/41316>
>
> I had previously added a sqlalchemy-migrate
> migration script to add an
> event_type table, and had that working, but then was
> asked to instead
> use Alembic for migrations. So, I removed the
> sqlalchemy-migrate
> migration file and added an Alembic migration [1].
>
> Unfortunately, I am getting the following error when
> running tests:
>
> OperationalError: (OperationalError) table
> event_type already exists
> u'\nCREATE TABLE event_type (\n\tid INTEGER NOT
> NULL, \n\t"desc"
> VARCHAR(255), \n\tPRIMARY KEY (id), \n\tUNIQUE
> ("desc")\n)\n\n' ()
>
> The migration adds the event_type table. I've seen
> this error occur
> before when using SQLite due to SQLite's ALTER TABLE
> statement not
> allowing the rename of a column. In the
> sqlalchemy-migrate migration, I
> had a specialized SQLite migration upgrade [2] and
> downgrade [3]
> script,
> but I'm not sure how I am supposed to handle this in
> Alembic. Could
> someone help me out?
>
> Thanks,
> -jay
>
> [1]
>
> https://review.openstack.org/#__/c/41316/16/ceilometer/__storage/sqlalchemy/
> <https://review.openstack.org/#/c/41316/16/ceilometer/storage/sqlalchemy/>
> alembic/versions/49036daaaafd___add_event_types.py
>
> [2]
>
> https://review.openstack.org/#__/c/41316/14/ceilometer/__storage/sqlalchemy/
> <https://review.openstack.org/#/c/41316/14/ceilometer/storage/sqlalchemy/>
> migrate_repo/versions/013___sqlite_upgrade.sql
>
> [3]
>
> https://review.openstack.org/#__/c/41316/14/ceilometer/__storage/sqlalchemy/
> <https://review.openstack.org/#/c/41316/14/ceilometer/storage/sqlalchemy/>
> migrate_repo/versions/013___sqlite_downgrade.sql
>
>
> _________________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.__org
> <mailto:OpenStack-dev at lists.openstack.org>
> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
>
>
> _________________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.__org
> <mailto:OpenStack-dev at lists.openstack.org>
> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
>
> _________________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.__org
> <mailto:OpenStack-dev at lists.openstack.org>
> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
>
>
> _________________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.__org
> <mailto:OpenStack-dev at lists.openstack.org>
> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
More information about the OpenStack-dev
mailing list