[openstack-dev] [neutron] Plethora of dbase migration questions...

Paul Michali pc at michali.net
Tue Jul 7 14:05:37 UTC 2015


Yes, I wasn't using the --service option, so I suspect that is why my
down_version was wrong.  In talking with Akihiro, I added a check to PEP8
and made sure that it fails if head is wrong. It is:
https://review.openstack.org/#/c/199082/ (of course that failed py27 - I've
got to see if there was some recent breakage in vpn repo, again).

Regarding the migration, one of the new columns may be None, but there must
be at least one IP version entry (there is an existing test in VPN for
using a router w/o an external IP set). Since the new code will rely on
these new fields, I'd like to populate them as part of the migration. I
think it would be more complicated to handle during operation.

Does anyone have examples of how to do queries of objects, from the
migration upgrade() code?


Regards,

PCM

On Tue, Jul 7, 2015 at 9:02 AM Akihiro Motoki <amotoki at gmail.com> wrote:

> 2015-07-07 21:39 GMT+09:00 Henry Gessau <gessau at cisco.com>:
>
>>  On Tue, Jul 07, 2015, Paul Michali <pc at michali.net> <pc at michali.net>
>> wrote:
>>
>> Thanks Salvatore for the responses. See @PCM in-line...
>>
>>
>>
>>  On Tue, Jul 7, 2015 at 6:14 AM Salvatore Orlando <sorlando at nicira.com>
>> wrote:
>>
>>> Some comments inline.
>>>
>>>  Salvatore
>>>
>>>    On 6 July 2015 at 20:00, Paul Michali < <pc at michali.net>
>>> pc at michali.net> wrote:
>>>
>>>> Hi,
>>>>
>>>>  I have some urgent requests about migration that I'm hoping to get
>>>> some info on. I'm working on a bug where I need to add two (related) fields
>>>> to a table for VPNaaS. Here's the objectives related to migration...
>>>>
>>>>  1) create local_v4_ip and lcoal_v6_ip fields in the vpnservice table
>>>> 2) for each entry in the vpnservice table:
>>>>     2.1) Get the router.gw_port.fixed_ips list
>>>>     2.2) Determine the version of each fixed IP and store the first of
>>>> each version (if any) into the appropriate new field.
>>>>
>>>>  I have created a migration file, and I changed the down_revision to
>>>> be the number of the revision that is the first in the migration chain in
>>>> the VPN repo.
>>>>
>>>>  Here are the many questions I have...
>>>>
>>>>  When I look in the VPN repo, the HEAD file has the version 'kilo',
>>>> which is not the current head.
>>>>
>>>
>>>>  Shouldn't it the version number of the first file in the migration
>>>> chain?
>>>>
>>>
>>>      It should indeed. How are you generating the revision script?
>>> Using neutron-db-manage it should be updated automatically [1]
>>>
>>
>>  @PCM I ran neutron-db-manage, when in the neutron repo, and it assigned
>> some version, but it was not the latest in the neutron-vpnaas repo.
>>
>> neutron-db-manage does not handle alembic branches in separate repos very
>> well at all yet. I am working on updating it with
>> <https://review.openstack.org/198524>https://review.openstack.org/198524
>> but I have quite a lot left to do.
>>
>
> Yes, at now we have implicit order of running alembic migrations.
> First run neutron db migration and then advanced service migrations.
>
> I do not fully understand how alembic branch mechanism works, but
> I think we can have a common ancestor and have multiple branches,
> and each branch can evolve independently.
>
>
>>
>>  I checked the VPN repo and there were a chain of versions, which I used
>> to determine what the head should be and have set the version accordingly.
>> However, in the current repo, head is set to "kilo", which appears to be
>> incorrect.  The versions are:
>>
>>  56893333aa52
>> kilo   <<< HEAD
>> 3ea02b2a773e
>> start_neutron_vpnaas
>> None
>>
>> Ouch. That is an error, because <https://review.openstack.org/190569>
>> https://review.openstack.org/190569 should have updated HEAD but didn't.
>>
>> The version sequence (you can see it in any devstack run) is:
>>
>> INFO  [alembic.migration] Running upgrade  -> start_neutron_vpnaas, start
>> neutron-vpnaas chain
>> INFO  [alembic.migration] Running upgrade start_neutron_vpnaas ->
>> 3ea02b2a773e, add_index_tenant_id
>> INFO  [alembic.migration] Running upgrade 3ea02b2a773e -> kilo, kilo
>> INFO  [alembic.migration] Running upgrade kilo -> 56893333aa52, fix
>> identifier map fk
>>
>
> It seems we don't have an appropriate check for HEAD revision in at least
> VPNaaS repo.
> Paul and I just discussed it. We need to improve the check too.
>
>
>> Should I do a separate commit that fixes the HEAD file, or just fix it as
>> part of the bug fix I'm working on.
>>
>> Yes, you should immediately submit a patch to change HEAD to 56893333aa52.
>>
>>
>>  BTW, at one point, after having correctly set the HEAD and versions in
>> my new migration file, I think I ran neutron-db-manage check_migration, and
>> I think it set the HEAD to my version, but it did that in the neutron repo,
>> and not the VPN repo.  I might have been running from the wrong repo?
>>
>> I working on updating the devref docs for this process. Things have
>> changed quite a bit with the alembic branches in separate repos.
>>
>>
>>
>>
>>>   For my commit, I'm assuming I change the HEAD file to use my
>>>> migration file's version?
>>>>
>>>
>>>      You can do that manually too, yes.
>>>
>>>
>>>>
>>>>  I set HEAD to my migration file, and my file has a down revision of
>>>> the previous head's revision. If I run 'neutron-db-manage --config-file
>>>> ../neutron/etc/neutron.conf --config-file
>>>> ../neutron/etc/neutron/plugins/ml2/ml2_conf.ini check_migration' there is
>>>> no output so I guess that is OK.
>>>>
>>>>  As I develop my new migration file, is there a way that I can test it
>>>> (running neutron-db-migration, maybe)?
>>>>
>>>
>>>      When I test migrations I usually dump the database, run the
>>> migration with neutron-db-manage upgrade HEAD (I think it's not necessary
>>> to specify HEAD), and restore the db from the dump if the migration fails.
>>>
>>>
>>>>  Is there a way to run the migration file under the debugger, as well
>>>> (importing pdb, for example)?
>>>>
>>>
>>>      The migration process is just like any python application, so I
>>> guess you can debug it with pdb.
>>>
>>
>>  @PCM Ah, so use "neutron-db-manage upgrade HEAD". That was the piece
>> that was missing. I take it there are no specific unit tests of the
>> migration files?
>>
>>
>>
>>>
>>>>
>>>>  In the migration, I can add the columns needed. What's the best way
>>>> to fill out those fields - using raw SQL queries or create a Session object
>>>> and access the VpnService object's router object?
>>>>
>>>
>>>      If the default value for the column is not enough, and you need to
>>> specify a value which depends on other values in the same row I would
>>> prefer plain SQL statements, but if that become cumbersome I guess it's ok
>>> to use sqlalchemy's session.
>>>
>>>
>>>>  I see there is some op.bind() call and then engine.execute(), but
>>>> could use some help on the best way to extract the needed queries (I need
>>>> to access the vpnservice's router, and then access the (Port) gw_port
>>>> relationship, and from that access the (IPAllocation) fixed_ips list).
>>>>
>>>
>>>      Perhaps you can point us to the review pages on gerrit, and we can
>>> provide detailed comments there.
>>>
>>
>>  @PCM Yeah, I haven't pushed it up yet. I have a few more changes to
>> make, and should be able to get it up in a few days. The LP bug is 1464387.
>>
>>  Essentially, in the vpnservices table, I'm adding an IPv4 and/or IPv6
>> addresses for the  "local" end of VPN connections that will be established.
>> This is to allow alternative VPN implementation (appliances, separate S/W,
>> H/W, VM based VPN, etc) to specify addresses different than what is
>> available on the Neutron router.
>>
>>  However, for the reference implementation, we'll use the Neutron
>> router's fixed_ips list (as is done today), and to handle the migration,
>> I'm thinking the following is needed:
>>
>>  1) create the new columns.
>> 2) Identify the router for that service and obtain it's GW fixed_ips list.
>> 3) Pick first IPv4 address (if any) and IPv6 address (if any), and store
>> in new columns.
>>
>>  So I need to form a query and code to do this.
>>
>>
>>
>>>
>>>>
>>>>  Appreciate any advise here on how to debug the migration stuff...
>>>>
>>>>  Paul Michali (pc_m)
>>>>
>>>
>>>      [1]
>>> http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124
>>>
>>>
>>>>
>>>>
>>>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150707/60093858/attachment.html>


More information about the OpenStack-dev mailing list