[openstack-dev] [Ceilometer] Need help with Alembic...
Alexei Kornienko
alexei.kornienko at gmail.com
Tue Aug 27 17:01:15 UTC 2013
Hello,
Please see my answers inline and my previous email regarding this topic:
> Hello,
>
> This conversion is actually quite simple. We are currently working to
> support alembic migrations in ceilometer:
> https://blueprints.launchpad.net/ceilometer/+spec/convert-to-alembic
>
> For now we agreed that conversion process will be the following:
> 1) Create folder for alembic migrations and configure engine correctly
> 2) Run alembic migrations after sqlalchemy-migrate (alembic creates a
> separate stamp table by default)
> 3) Create new migrations in alembic
> 4) Sync model with migrations* -
> https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-db-sync-models-with-migrations
>
> 5) Remove old migration files after the end of support period (2
> releases)
>
> * Is needed to remove the need of base migration so the clean database
> could be created from models directly without the need of migrations.
> This allows to simply drop old migrations without the need to compact
> them to one big migration (133_folsom.py in nova for example)
>
> Please share your thoughts about proposed process.
>
> Regards,
> Alexei Kornienko
Regards,
Alexei Kornienko
On 08/27/2013 07:30 PM, Jay Pipes wrote:
> On 08/27/2013 04:32 AM, Boris Pavlovic wrote:
>> Jay,
>>
>> I should probably share to you about our work around DB.
>>
>> Migrations should be run only in production and only for production
>> backends (e.g. psql and mysql)
>> In tests we should use Schemas created by Models
>> (BASE.metadata.create_all())
>
> Agree on both.
>
>> We are not able to use in this approach in moment because we don't have
>> any mechanism to check that MODELS and SCHEMAS are EQUAL.
>> And actually MODELS and SCHEMAS are DIFFERENT.
>
> Sorry, I don't understand the connection... how does not having a
> codified way of determining the difference between model and schema
> (BTW, this does exist in sqlalchemy-migrate... look at the
> compare_model_to_db method) not allow you to use metadata.create_all()
> in tests or mean that you can't run migrations only in production?
As Boris said we'll use 2 completely different ways to create DB schema
in production and test environment. Cause of this we won't be able to
guarantee that code is correct unless we'll have a dedicated test that
will assure that we work with the same DB schema in both envs.
>
>> E.g. in Celiometer we have BP that syncs models and migration
>> https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-db-sync-models-with-migrations
>>
>> (in other projects we are doing the same)
>>
>> And also we are working around (oslo) generic tests that checks that
>> models and migrations are equal:
>> https://review.openstack.org/#/c/42307/
>
> OK, cool.
>
>> So in our roadmap (in this case is):
>> 1) Soft switch to alembic (with code that allows to have sqla-migrate
>> and alembic migration in the same time)
>
> I don't see the point in this at all... I would rather see patches
> that just switch to Alembic and get rid of SQLAlchemy-migrate. Create
> an initial Alembic migration that has the last state of the database
> schema under SQLAlchemy-migrate... and then delete SA-Migrate.
It's not that simple as it seems. Please take into account that we have
to keep SA-migrate + migrations during maintenance period for all
projects. Cause of this we have to go the long way and keep both engines
running before we'll be able to completely remove SA-migrate.
>
>> 2) Sync Models and Migrations (fix DB schemas also)
>> 3) Add from oslo generic test that checks all this stuff
>> 4) Use BASE.create_all() for Schema creation instead of migrations.
>
> This is already done in some projects, IIRC... (Glance used to be this
> way, at least)
>
>> But in OpenStack is not so simple to implement such huge changes, so it
>> take some time=)
>>
>>
>> Best regards,
>> Boris Pavlovic
>> ---
>> Mirantis Inc.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Aug 27, 2013 at 12:02 AM, Jay Pipes <jaypipes at gmail.com
>> <mailto:jaypipes at gmail.com>> wrote:
>>
>> On 08/26/2013 03:40 PM, Herndon, John Luke (HPCS - Ft. Collins)
>> wrote:
>>
>> Jay -
>>
>> It looks there is an error in the migration script that causes
>> it to abort:
>>
>> AttributeError: 'ForeignKeyConstraint' object has no attribute
>> 'drop'
>>
>> My guess is the migration runs on the first test, creates event
>> types
>> table fine, but exits with the above error, so migration is not
>> "complete". Thus every subsequent test tries to migrate the
>> db, and
>> notices that event types already exists.
>>
>>
>> I'd corrected that particular mistake and pushed an updated
>> migration script.
>>
>> Best,
>> -jay
>>
>>
>>
>> -john
>>
>> On 8/26/13 1:15 PM, "Jay Pipes" <jaypipes at gmail.com
>> <mailto:jaypipes at gmail.com>> wrote:
>>
>> I just noticed that every single test case for SQL-driver
>> storage is
>> executing every single migration upgrade before every single
>> test case
>> run:
>>
>> https://github.com/openstack/__ceilometer/blob/master/__ceilometer/tests/db.py
>> <https://github.com/openstack/ceilometer/blob/master/ceilometer/tests/db.py>
>> #L46
>>
>> https://github.com/openstack/__ceilometer/blob/master/__ceilometer/storage/imp
>> <https://github.com/openstack/ceilometer/blob/master/ceilometer/storage/imp>
>> l_sqlalchemy.py#L153
>>
>> instead of simply creating a new database schema from the
>> models in the
>> current source code base using a call to
>> sqlalchemy.MetaData.create___all().
>>
>> This results in re-running migrations over and over again,
>> instead of
>> having dedicated migration tests that would test each
>> migration
>> individually, as is the case in projects like Glance...
>>
>> Is this intentional?
>>
>> Best,
>> -jay
>>
>> On 08/26/2013 02:59 PM, Sandy Walsh wrote:
>>
>> I'm getting the same problem with a different migration
>> (mine is
>> complaining that a column already exists)
>>
>> http://paste.openstack.org/__show/44512/
>> <http://paste.openstack.org/show/44512/>
>>
>> I've compared it to the other migrations and it seems
>> fine.
>>
>> -S
>>
>> On 08/26/2013 02:34 PM, Jay Pipes wrote:
>>
>> Hey all,
>>
>> I'm trying to figure out what is going wrong with my
>> code for this
>> patch:
>>
>> https://review.openstack.org/__41316
>> <https://review.openstack.org/41316>
>>
>> I had previously added a sqlalchemy-migrate
>> migration script to add an
>> event_type table, and had that working, but then was
>> asked to instead
>> use Alembic for migrations. So, I removed the
>> sqlalchemy-migrate
>> migration file and added an Alembic migration [1].
>>
>> Unfortunately, I am getting the following error when
>> running tests:
>>
>> OperationalError: (OperationalError) table
>> event_type already exists
>> u'\nCREATE TABLE event_type (\n\tid INTEGER NOT
>> NULL, \n\t"desc"
>> VARCHAR(255), \n\tPRIMARY KEY (id), \n\tUNIQUE
>> ("desc")\n)\n\n' ()
>>
>> The migration adds the event_type table. I've seen
>> this error occur
>> before when using SQLite due to SQLite's ALTER TABLE
>> statement not
>> allowing the rename of a column. In the
>> sqlalchemy-migrate migration, I
>> had a specialized SQLite migration upgrade [2] and
>> downgrade [3]
>> script,
>> but I'm not sure how I am supposed to handle this in
>> Alembic. Could
>> someone help me out?
>>
>> Thanks,
>> -jay
>>
>> [1]
>>
>> https://review.openstack.org/#__/c/41316/16/ceilometer/__storage/sqlalchemy/
>> <https://review.openstack.org/#/c/41316/16/ceilometer/storage/sqlalchemy/>
>> alembic/versions/49036daaaafd___add_event_types.py
>>
>> [2]
>>
>> https://review.openstack.org/#__/c/41316/14/ceilometer/__storage/sqlalchemy/
>> <https://review.openstack.org/#/c/41316/14/ceilometer/storage/sqlalchemy/>
>> migrate_repo/versions/013___sqlite_upgrade.sql
>>
>> [3]
>>
>> https://review.openstack.org/#__/c/41316/14/ceilometer/__storage/sqlalchemy/
>> <https://review.openstack.org/#/c/41316/14/ceilometer/storage/sqlalchemy/>
>> migrate_repo/versions/013___sqlite_downgrade.sql
>>
>>
>> _________________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.__org
>> <mailto:OpenStack-dev at lists.openstack.org>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>>
>> _________________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.__org
>> <mailto:OpenStack-dev at lists.openstack.org>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>> _________________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.__org
>> <mailto:OpenStack-dev at lists.openstack.org>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>>
>> _________________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.__org
>> <mailto:OpenStack-dev at lists.openstack.org>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
More information about the OpenStack-dev
mailing list