Open Stack

Hello,

This conversion is actually quite simple. We are currently working to 
support alembic migrations in ceilometer:
https://blueprints.launchpad.net/ceilometer/+spec/convert-to-alembic

For now we agreed that conversion process will be the following:
1) Create folder for alembic migrations and configure engine correctly
2) Run alembic migrations after sqlalchemy-migrate (alembic creates a 
separate stamp table by default)
3) Create new migrations in alembic
4) Sync model with migrations* - 
https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-db-sync-models-with-migrations
5) Remove old migration files after the end of support period (2 releases)

* Is needed to remove the need of base migration so the clean database 
could be created from models directly without the need of migrations.
This allows to simply drop old migrations without the need to compact 
them to one big migration (133_folsom.py in nova for example)

Please share your thoughts about proposed process.

Regards,
Alexei Kornienko

On 07/29/2013 02:13 AM, Jamie Lennox wrote:
>
>
>
> ----- Original Message -----
>> From: "Doug Hellmann" <doug.hellmann at dreamhost.com>
>> To: "OpenStack Development Mailing List" <openstack-dev at lists.openstack.org>
>> Sent: Saturday, 27 July, 2013 4:15:53 AM
>> Subject: Re: [openstack-dev] [Keystone] Alembic support
>>
>>
>>
>>
>> On Fri, Jul 26, 2013 at 2:04 PM, Adam Young < ayoung at redhat.com > wrote:
>>
>>
>>
>> On 07/26/2013 01:55 PM, Doug Hellmann wrote:
>>
>>
>>
>> It makes sense, though, if the data is going to be (potentially) coming from
>> separate databases. Is that any different for sqlalchemy-migrate?
>>
>> As far as tracking the separate migrations for the different schemas, that's
>> handled by running "alembic init" for each schema to create a different
>> environment. The environments should then have unique "version_table" values
>> defined in the alembic.ini that the versions of each schema can be tracked
>> separately. I suggest alembic_version_${schema_owner} where schema_owner is
>> the subset of the schema (i.e., policy, tokens, or identity), or the
>> extension name.
>>
>> It think that this will require enough of a change that I would like to do it
>> in Icehouse, and have a detailed blueprint written up for it.
>>
>>
>> That seems reasonable. Splitting one database into 3 (or more) will take some
>> consideration.
> +1 I think we can leave this till Icehouse, though continue discussing the how.
> I'm interested to know how the changeover will work.
>
>> Doug
>>
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Jul 26, 2013 at 1:42 PM, Dolph Mathews < dolph.mathews at gmail.com >
>> wrote:
>>
>>
>>
>> Based on the docs, it looks like you need to start with separate sqlalchemy
>> engines with their own metadata instances for each environment you want to
>> migrate. That sounds like a significant refactor from where we are today
>> (everything shares keystone.common.sql.core.ModelBase).
>>
>>
>> On Thu, Jul 25, 2013 at 10:41 PM, Morgan Fainberg < m at metacloud.com > wrote:
>>
>>
>> +1 to getting the multiple repos in place. Moving to Alembric later on in H
>> or even as the first commit of I should meet our goals to be on Alembric in
>> a reasonable timeframe. This also allows us to ensure we aren't rushing the
>> work to get our migration repos over to Alembric.
>>
>> I think that allowing the extensions to have their own repos sooner is
>> better, and if we end up with an extension that has more than 1 or 2
>> migrations, we have probably accepted code that is far from fully baked (and
>> we should evaluate how that code made it in).
>>
>> I am personally in favor of making the first commit of Icehouse (barring any
>> major issue) the point in which we move to Alembric. We can be selective in
>> taking extension modifications that add migration repos if it is a major
>> concern that moving to Alembric is going to be really painful.
>>
>> Cheers,
>> Morgan Fainberg
>>
>> On Thu, Jul 25, 2013 at 7:35 PM, Adam Young < ayoung at redhat.com > wrote:
>>
>>
>> I've been looking into Alembic support. It seems that there is one thing
>> missing that I was counting on: multiple migration repos. It might be
>> supported, but the docs are thin, and reports vary.
>>
>> In the current Keystone implementation, we have a table like this:
>> mysql> desc migrate_version;
>> +-----------------+--------------+------+-----+---------+-------+
>> | Field | Type | Null | Key | Default | Extra |
>> +-----------------+--------------+------+-----+---------+-------+
>> | repository_id | varchar(250) | NO | PRI | NULL | |
>> | repository_path | text | YES | | NULL | |
>> | version | int(11) | YES | | NULL | |
>> +-----------------+--------------+------+-----+---------+-------+
>>
>>
>> Right now we only have one row in there:
>>
>> keystone | /opt/stack/keystone/keystone/common/sql/migrate_repo | 0
>>
>>
>> However, we have been lumping all of our migrations together into a singel
>> repo, and we are just now looking to sort them out. For example, Policy,
>> Tokens, and Identity do not really need to share a database. As such, they
>> could go into separate migration repos, and it would keep changes to one
>> from stepping on changes to another, and avoiding the continuous rebasing
>> problem we currently have.
>>
>> In addition, we want to put each of the extensions into their own repos. This
>> happens to be an important time for that, as we have three extensions coming
>> in that need SQL repos: OAuth, KDS, and Attribute Mapping.
>>
>> I think we should delay moving Keystone to Alembic until the end of Havana,
>> or as the first commit in Icehouse. That way, we have a clean cut over
>> point. We can decide then whether to backport the Extnesion migrations or
>> leave them under sql_alchemy migrations. Mixing the two technologies side by
>> side for a short period of time is going to be required, and I think we need
>> to have a clear approach in place to avoid a mess.
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>
>> --
>>
>> -Dolph
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev