Open Stack

On Sun, Jul 14, 2013 at 12:29 PM, Boris Pavlovic <boris at pavlovic.me> wrote:

> Hi Dolph,
>
> Yes, I agree that there are some problems with sqlite and migrations.
> But I am not agree with approach of fully removing sqlite.
>
> It is pretty useful for testing.
>

To be clear, I don't have any interest in dropping support for running
keystone against sqlite as backend for development / testing. It's
certainly useful for running functional tests in-memory. However, such
databases don't need to be created by db_sync or using any migrations
whatsoever.

It's the migrations themselves where I want to drop support for sqlite. No
one is migrating production data in a sqlite database.


>
> So the right approach that we are trying to provide in whole OpenStack is
> next:
>
> 1) Use alembic as migration tool (don't support sqlite in migrations)
>

In my original email, I failed to cite where I got the idea (because I
couldn't find a link, perhaps someone else knows of it?), but the same
concern was expressed by some alembic-related documentation or blog post
that I had recently read. I don't expect to see keystone making much
progress towards switching to alembic until icehouse or even later. In the
mean time, I'd like to do what we can to limit our pain while working with
sqlaclhemy-migrate (which mostly involves writing entirely different code
paths to handle sqlite's lack of support for proper schema evolution).


> 2) Test migrations in two ways:
> a) run all migrations with real data against supported backends (mysql,
> psql)
> b) test that models and migrations are synced in all backends (mysql, psql)
>

+1


> 3) Unit tests should be run against DB that was created from models (not
> from migrations)
>

+1


> 4) Unit tests should support all backends (sqlite also)
>

+1


>
>
> If you are interested in this sphere I could try to describe current state
> in more words.
>

I think we're already in complete agreement! I just wasn't clear in my
original email (apologies!).


>
>
> Best regards,
> Boris Pavlovic
>
>
>
> On Sat, Jul 13, 2013 at 12:19 AM, Monty Taylor <mordred at inaugust.com>wrote:
>
>>
>>
>> On 07/11/2013 01:12 PM, Dolph Mathews wrote:
>> > Just as a general statement, outside the scope of openstack, I don't
>> > think sqlite is intended to support schema evolution. From the sqlite
>> > docs [1]: "SQLite supports a limited subset of ALTER TABLE. [...] It is
>> > not possible to rename a column, remove a column, or add or remove
>> > constraints from a table."
>> >
>> > We've been through hell trying to support migrations on sqlite, because
>> > we test against sqlite, and because we test our migrations... on sqlite.
>> > So, we've already shot ourselves in the foot. We're clearly moving
>> > towards gating against mysql + postgresql, so in the mean time, let's
>> > limit the amount of effort we put into further support sqlite migrations
>> > until we can safely rip it out altogether.
>> >
>> > [1]: http://www.sqlite.org/lang_altertable.html
>>
>> I agree. The reason to use sqlite in unitests and stuff is because it's
>> easy and doesn't require users and system things and everything. If
>> we're spending extra effort to maintain the simple thing, then it's
>> probably not a simple thing.
>>
>> As an aside, (ignore the fact that I'm a former Drizzle core dev) it
>> might be worthwhile taking 30 minutes one day and exploring a drizzle
>> database test fixture. One of the things we did in drizzle was make it
>> not need any bootstrapping and to work sanely with no config files ...
>> so launching a drizzle on a spare port, running database tests against
>> it and then deleting it should actually be super simple - and at the
>> worst no harder than doing what glance does in their functional tests.
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 

-Dolph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130715/7fa283b0/attachment.html>