[openstack-dev] [Ceilometer] Need help with Alembic...

Alexei Kornienko alexei.kornienko at gmail.com
Tue Aug 27 17:01:15 UTC 2013


Hello,

Please see my answers inline and my previous email regarding this topic:
> Hello,
>
> This conversion is actually quite simple. We are currently working to 
> support alembic migrations in ceilometer:
> https://blueprints.launchpad.net/ceilometer/+spec/convert-to-alembic
>
> For now we agreed that conversion process will be the following:
> 1) Create folder for alembic migrations and configure engine correctly
> 2) Run alembic migrations after sqlalchemy-migrate (alembic creates a 
> separate stamp table by default)
> 3) Create new migrations in alembic
> 4) Sync model with migrations* - 
> https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-db-sync-models-with-migrations 
>
> 5) Remove old migration files after the end of support period (2 
> releases)
>
> * Is needed to remove the need of base migration so the clean database 
> could be created from models directly without the need of migrations.
> This allows to simply drop old migrations without the need to compact 
> them to one big migration (133_folsom.py in nova for example)
>
> Please share your thoughts about proposed process.
>
> Regards,
> Alexei Kornienko 

Regards,
Alexei Kornienko

On 08/27/2013 07:30 PM, Jay Pipes wrote:
> On 08/27/2013 04:32 AM, Boris Pavlovic wrote:
>> Jay,
>>
>> I should probably share to you about our work around DB.
>>
>> Migrations should be run only in production and only for production
>> backends (e.g. psql and mysql)
>> In tests we should use Schemas created by Models
>> (BASE.metadata.create_all())
>
> Agree on both.
>
>> We are not able to use in this approach in moment  because we don't have
>> any mechanism to check that MODELS and SCHEMAS are EQUAL.
>> And actually MODELS and SCHEMAS are DIFFERENT.
>
> Sorry, I don't understand the connection... how does not having a 
> codified way of determining the difference between model and schema 
> (BTW, this does exist in sqlalchemy-migrate... look at the 
> compare_model_to_db method) not allow you to use metadata.create_all() 
> in tests or mean that you can't run migrations only in production?
As Boris said we'll use 2 completely different ways to create DB schema 
in production and test environment. Cause of this we won't be able to 
guarantee that code is correct unless we'll have a dedicated test that 
will assure that we work with the same DB schema in both envs.
>
>> E.g. in Celiometer we have BP that syncs models and migration
>> https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-db-sync-models-with-migrations 
>>
>> (in other projects we are doing the same)
>>
>> And also we are working around (oslo) generic tests that checks that
>> models and migrations are equal:
>> https://review.openstack.org/#/c/42307/
>
> OK, cool.
>
>> So in our roadmap (in this case is):
>> 1) Soft switch to alembic (with code that allows to have sqla-migrate
>> and alembic migration in the same time)
>
> I don't see the point in this at all... I would rather see patches 
> that just switch to Alembic and get rid of SQLAlchemy-migrate. Create 
> an initial Alembic migration that has the last state of the database 
> schema under SQLAlchemy-migrate... and then delete SA-Migrate.
It's not that simple as it seems. Please take into account that we have 
to keep SA-migrate + migrations during maintenance period for all 
projects. Cause of this we have to go the long way and keep both engines 
running before we'll be able to completely remove SA-migrate.

>
>> 2) Sync Models and Migrations (fix DB schemas also)
>> 3) Add from oslo generic test that checks all this stuff
>> 4) Use BASE.create_all() for Schema creation instead of migrations.
>
> This is already done in some projects, IIRC... (Glance used to be this 
> way, at least)
>
>> But in OpenStack is not so simple to implement such huge changes, so it
>> take some time=)
>>
>>
>> Best regards,
>> Boris Pavlovic
>> ---
>> Mirantis Inc.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Aug 27, 2013 at 12:02 AM, Jay Pipes <jaypipes at gmail.com
>> <mailto:jaypipes at gmail.com>> wrote:
>>
>>     On 08/26/2013 03:40 PM, Herndon, John Luke (HPCS - Ft. Collins) 
>> wrote:
>>
>>         Jay -
>>
>>         It looks there is an error in the migration script that causes
>>         it to abort:
>>
>>         AttributeError: 'ForeignKeyConstraint' object has no attribute
>>         'drop'
>>
>>         My guess is the migration runs on the first test, creates event
>>         types
>>         table fine, but exits with the above error, so migration is not
>>         "complete". Thus every subsequent test tries to migrate the 
>> db, and
>>         notices that event types already exists.
>>
>>
>>     I'd corrected that particular mistake and pushed an updated
>>     migration script.
>>
>>     Best,
>>     -jay
>>
>>
>>
>>         -john
>>
>>         On 8/26/13 1:15 PM, "Jay Pipes" <jaypipes at gmail.com
>>         <mailto:jaypipes at gmail.com>> wrote:
>>
>>             I just noticed that every single test case for SQL-driver
>>             storage is
>>             executing every single migration upgrade before every single
>>             test case
>>             run:
>>
>> https://github.com/openstack/__ceilometer/blob/master/__ceilometer/tests/db.py
>> <https://github.com/openstack/ceilometer/blob/master/ceilometer/tests/db.py>
>>             #L46
>>
>> https://github.com/openstack/__ceilometer/blob/master/__ceilometer/storage/imp
>> <https://github.com/openstack/ceilometer/blob/master/ceilometer/storage/imp>
>>             l_sqlalchemy.py#L153
>>
>>             instead of simply creating a new database schema from the
>>             models in the
>>             current source code base using a call to
>>             sqlalchemy.MetaData.create___all().
>>
>>             This results in re-running migrations over and over again,
>>             instead of
>>             having dedicated migration tests that would test each 
>> migration
>>             individually, as is the case in projects like Glance...
>>
>>             Is this intentional?
>>
>>             Best,
>>             -jay
>>
>>             On 08/26/2013 02:59 PM, Sandy Walsh wrote:
>>
>>                 I'm getting the same problem with a different migration
>>                 (mine is
>>                 complaining that a column already exists)
>>
>>                 http://paste.openstack.org/__show/44512/
>>                 <http://paste.openstack.org/show/44512/>
>>
>>                 I've compared it to the other migrations and it seems 
>> fine.
>>
>>                 -S
>>
>>                 On 08/26/2013 02:34 PM, Jay Pipes wrote:
>>
>>                     Hey all,
>>
>>                     I'm trying to figure out what is going wrong with my
>>                     code for this
>>                     patch:
>>
>>                     https://review.openstack.org/__41316
>>                     <https://review.openstack.org/41316>
>>
>>                     I had previously added a sqlalchemy-migrate
>>                     migration script to add an
>>                     event_type table, and had that working, but then was
>>                     asked to instead
>>                     use Alembic for migrations. So, I removed the
>>                     sqlalchemy-migrate
>>                     migration file and added an Alembic migration [1].
>>
>>                     Unfortunately, I am getting the following error when
>>                     running tests:
>>
>>                     OperationalError: (OperationalError) table
>>                     event_type already exists
>>                     u'\nCREATE TABLE event_type (\n\tid INTEGER NOT
>>                     NULL, \n\t"desc"
>>                     VARCHAR(255), \n\tPRIMARY KEY (id), \n\tUNIQUE
>>                     ("desc")\n)\n\n' ()
>>
>>                     The migration adds the event_type table. I've seen
>>                     this error occur
>>                     before when using SQLite due to SQLite's ALTER TABLE
>>                     statement not
>>                     allowing the rename of a column. In the
>>                     sqlalchemy-migrate migration, I
>>                     had a specialized SQLite migration upgrade [2] and
>>                     downgrade [3]
>>                     script,
>>                     but I'm not sure how I am supposed to handle this in
>>                     Alembic. Could
>>                     someone help me out?
>>
>>                     Thanks,
>>                     -jay
>>
>>                     [1]
>>
>> https://review.openstack.org/#__/c/41316/16/ceilometer/__storage/sqlalchemy/
>> <https://review.openstack.org/#/c/41316/16/ceilometer/storage/sqlalchemy/>
>> alembic/versions/49036daaaafd___add_event_types.py
>>
>>                     [2]
>>
>> https://review.openstack.org/#__/c/41316/14/ceilometer/__storage/sqlalchemy/
>> <https://review.openstack.org/#/c/41316/14/ceilometer/storage/sqlalchemy/>
>> migrate_repo/versions/013___sqlite_upgrade.sql
>>
>>                     [3]
>>
>> https://review.openstack.org/#__/c/41316/14/ceilometer/__storage/sqlalchemy/
>> <https://review.openstack.org/#/c/41316/14/ceilometer/storage/sqlalchemy/>
>> migrate_repo/versions/013___sqlite_downgrade.sql
>>
>>
>> _________________________________________________
>>                     OpenStack-dev mailing list
>>                     OpenStack-dev at lists.openstack.__org
>> <mailto:OpenStack-dev at lists.openstack.org>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>>
>>             _________________________________________________
>>             OpenStack-dev mailing list
>>             OpenStack-dev at lists.openstack.__org
>>             <mailto:OpenStack-dev at lists.openstack.org>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>>             _________________________________________________
>>             OpenStack-dev mailing list
>>             OpenStack-dev at lists.openstack.__org
>>             <mailto:OpenStack-dev at lists.openstack.org>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>>
>>     _________________________________________________
>>     OpenStack-dev mailing list
>>     OpenStack-dev at lists.openstack.__org
>>     <mailto:OpenStack-dev at lists.openstack.org>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev 
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




More information about the OpenStack-dev mailing list