[openstack-dev] [keystone] sqlite doesn't support migrations

Doug Hellmann doug.hellmann at dreamhost.com
Mon Jul 15 18:40:37 UTC 2013


On Mon, Jul 15, 2013 at 11:58 AM, Dolph Mathews <dolph.mathews at gmail.com>wrote:

>
> On Sun, Jul 14, 2013 at 12:29 PM, Boris Pavlovic <boris at pavlovic.me>wrote:
>
>> Hi Dolph,
>>
>> Yes, I agree that there are some problems with sqlite and migrations.
>> But I am not agree with approach of fully removing sqlite.
>>
>> It is pretty useful for testing.
>>
>
> To be clear, I don't have any interest in dropping support for running
> keystone against sqlite as backend for development / testing. It's
> certainly useful for running functional tests in-memory. However, such
> databases don't need to be created by db_sync or using any migrations
> whatsoever.
>
> It's the migrations themselves where I want to drop support for sqlite. No
> one is migrating production data in a sqlite database.
>
>
>>
>> So the right approach that we are trying to provide in whole OpenStack is
>> next:
>>
>> 1) Use alembic as migration tool (don't support sqlite in migrations)
>>
>
> In my original email, I failed to cite where I got the idea (because I
> couldn't find a link, perhaps someone else knows of it?), but the same
> concern was expressed by some alembic-related documentation or blog post
> that I had recently read. I don't expect to see keystone making much
> progress towards switching to alembic until icehouse or even later. In the
> mean time, I'd like to do what we can to limit our pain while working with
> sqlaclhemy-migrate (which mostly involves writing entirely different code
> paths to handle sqlite's lack of support for proper schema evolution).
>

>From https://bitbucket.org/zzzeek/alembic:

"""

   - Don't break our necks over SQLite's inability to ALTER things. SQLite
   has almost no support for table or column alteration, and this is likely
   intentional. Alembic's design is kept simple by not contorting its core API
   around these limitations, understanding that SQLite is simply not intended
   to support schema changes. While Alembic's architecture can support
   SQLite's workarounds, and we will support these features provided someone
   takes the initiative to implement and test, until the SQLite developers
   decide to provide a fully working version of ALTER, it's still vastly
   preferable to use Alembic, or any migrations tool, with databases that are
   designed to work under the assumption of in-place schema migrations taking
   place.

"""

Moving to alembic still provides benefits (merging changesets with
migration scripts is easier and that the tool is actively maintained), but
migrations in SQLite isn't one of them.

Doug



>
>
>> 2) Test migrations in two ways:
>> a) run all migrations with real data against supported backends (mysql,
>> psql)
>> b) test that models and migrations are synced in all backends (mysql,
>> psql)
>>
>
> +1
>
>
>> 3) Unit tests should be run against DB that was created from models (not
>> from migrations)
>>
>
> +1
>
>
>> 4) Unit tests should support all backends (sqlite also)
>>
>
> +1
>
>
>>
>>
>> If you are interested in this sphere I could try to describe current
>> state in more words.
>>
>
> I think we're already in complete agreement! I just wasn't clear in my
> original email (apologies!).
>
>
>>
>>
>> Best regards,
>> Boris Pavlovic
>>
>>
>>
>> On Sat, Jul 13, 2013 at 12:19 AM, Monty Taylor <mordred at inaugust.com>wrote:
>>
>>>
>>>
>>> On 07/11/2013 01:12 PM, Dolph Mathews wrote:
>>> > Just as a general statement, outside the scope of openstack, I don't
>>> > think sqlite is intended to support schema evolution. From the sqlite
>>> > docs [1]: "SQLite supports a limited subset of ALTER TABLE. [...] It is
>>> > not possible to rename a column, remove a column, or add or remove
>>> > constraints from a table."
>>> >
>>> > We've been through hell trying to support migrations on sqlite, because
>>> > we test against sqlite, and because we test our migrations... on
>>> sqlite.
>>> > So, we've already shot ourselves in the foot. We're clearly moving
>>> > towards gating against mysql + postgresql, so in the mean time, let's
>>> > limit the amount of effort we put into further support sqlite
>>> migrations
>>> > until we can safely rip it out altogether.
>>> >
>>> > [1]: http://www.sqlite.org/lang_altertable.html
>>>
>>> I agree. The reason to use sqlite in unitests and stuff is because it's
>>> easy and doesn't require users and system things and everything. If
>>> we're spending extra effort to maintain the simple thing, then it's
>>> probably not a simple thing.
>>>
>>> As an aside, (ignore the fact that I'm a former Drizzle core dev) it
>>> might be worthwhile taking 30 minutes one day and exploring a drizzle
>>> database test fixture. One of the things we did in drizzle was make it
>>> not need any bootstrapping and to work sanely with no config files ...
>>> so launching a drizzle on a spare port, running database tests against
>>> it and then deleting it should actually be super simple - and at the
>>> worst no harder than doing what glance does in their functional tests.
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
>
> -Dolph
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130715/466e9b7c/attachment-0001.html>


More information about the OpenStack-dev mailing list