Open Stack

On 7 July 2015 at 14:00, Paul Michali <pc at michali.net> wrote:

> Thanks Salvatore for the responses. See @PCM in-line...
>
>
>
> On Tue, Jul 7, 2015 at 6:14 AM Salvatore Orlando <sorlando at nicira.com>
> wrote:
>
>> Some comments inline.
>>
>> Salvatore
>>
>> On 6 July 2015 at 20:00, Paul Michali <pc at michali.net> wrote:
>>
>>> Hi,
>>>
>>> I have some urgent requests about migration that I'm hoping to get some
>>> info on. I'm working on a bug where I need to add two (related) fields to a
>>> table for VPNaaS. Here's the objectives related to migration...
>>>
>>> 1) create local_v4_ip and lcoal_v6_ip fields in the vpnservice table
>>> 2) for each entry in the vpnservice table:
>>>     2.1) Get the router.gw_port.fixed_ips list
>>>     2.2) Determine the version of each fixed IP and store the first of
>>> each version (if any) into the appropriate new field.
>>>
>>> I have created a migration file, and I changed the down_revision to be
>>> the number of the revision that is the first in the migration chain in the
>>> VPN repo.
>>>
>>> Here are the many questions I have...
>>>
>>> When I look in the VPN repo, the HEAD file has the version 'kilo', which
>>> is not the current head.
>>>
>>
>>> Shouldn't it the version number of the first file in the migration chain?
>>>
>>
>> It should indeed. How are you generating the revision script? Using
>> neutron-db-manage it should be updated automatically [1]
>>
>
> @PCM I ran neutron-db-manage, when in the neutron repo, and it assigned
> some version, but it was not the latest in the neutron-vpnaas repo.
>

when you create a revision Alembic automatically assigns it a unique id.
However, the neutron migration CLI (neutron-db-manage) then should take
care of updating the HEAD file automatically. If this is not happening,
that's where the problem should be.


> I checked the VPN repo and there were a chain of versions, which I used to
> determine what the head should be and have set the version accordingly.
> However, in the current repo, head is set to "kilo", which appears to be
> incorrect.  The versions are:
>
> 56893333aa52
> kilo   <<< HEAD
> 3ea02b2a773e
> start_neutron_vpnaas
> None
>
> Should I do a separate commit that fixes the HEAD file, or just fix it as
> part of the bug fix I'm working on.
>

In order to pass functional tests the HEAD file must point to the topmost
revision (56893333aa52)


> BTW, at one point, after having correctly set the HEAD and versions in my
> new migration file, I think I ran neutron-db-manage check_migration, and I
> think it set the HEAD to my version, but it did that in the neutron repo,
> and not the VPN repo.  I might have been running from the wrong repo?
>

Yes, probably.
neutron-db-manage by default works on the neutron repo. In order to work
with a service repo you should specify it on the command line (
http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n41
).
This might also explain why the HEAD is not getting updated in your repo.


>
>
>
>> For my commit, I'm assuming I change the HEAD file to use my migration
>>> file's version?
>>>
>>
>> You can do that manually too, yes.
>>
>>
>>>
>>> I set HEAD to my migration file, and my file has a down revision of the
>>> previous head's revision. If I run 'neutron-db-manage --config-file
>>> ../neutron/etc/neutron.conf --config-file
>>> ../neutron/etc/neutron/plugins/ml2/ml2_conf.ini check_migration' there is
>>> no output so I guess that is OK.
>>>
>>> As I develop my new migration file, is there a way that I can test it
>>> (running neutron-db-migration, maybe)?
>>>
>>
>> When I test migrations I usually dump the database, run the migration
>> with neutron-db-manage upgrade HEAD (I think it's not necessary to specify
>> HEAD), and restore the db from the dump if the migration fails.
>>
>>
>>> Is there a way to run the migration file under the debugger, as well
>>> (importing pdb, for example)?
>>>
>>
>> The migration process is just like any python application, so I guess you
>> can debug it with pdb.
>>
>
> @PCM Ah, so use "neutron-db-manage upgrade HEAD". That was the piece that
> was missing. I take it there are no specific unit tests of the migration
> files?
>

... and also specify --service vpnaas

>
>
>
>>
>>>
>>> In the migration, I can add the columns needed. What's the best way to
>>> fill out those fields - using raw SQL queries or create a Session object
>>> and access the VpnService object's router object?
>>>
>>
>> If the default value for the column is not enough, and you need to
>> specify a value which depends on other values in the same row I would
>> prefer plain SQL statements, but if that become cumbersome I guess it's ok
>> to use sqlalchemy's session.
>>
>>
>>> I see there is some op.bind() call and then engine.execute(), but could
>>> use some help on the best way to extract the needed queries (I need to
>>> access the vpnservice's router, and then access the (Port) gw_port
>>> relationship, and from that access the (IPAllocation) fixed_ips list).
>>>
>>
>> Perhaps you can point us to the review pages on gerrit, and we can
>> provide detailed comments there.
>>
>
> @PCM Yeah, I haven't pushed it up yet. I have a few more changes to make,
> and should be able to get it up in a few days. The LP bug is 1464387.
>
> Essentially, in the vpnservices table, I'm adding an IPv4 and/or IPv6
> addresses for the  "local" end of VPN connections that will be established.
> This is to allow alternative VPN implementation (appliances, separate S/W,
> H/W, VM based VPN, etc) to specify addresses different than what is
> available on the Neutron router.
>
> However, for the reference implementation, we'll use the Neutron router's
> fixed_ips list (as is done today), and to handle the migration, I'm
> thinking the following is needed:
>
> 1) create the new columns.
> 2) Identify the router for that service and obtain it's GW fixed_ips list.
> 3) Pick first IPv4 address (if any) and IPv6 address (if any), and store
> in new columns.
>
> So I need to form a query and code to do this.
>
>
You can put all this logic in the DB migration and it should work, from
what I can tell.
An alternative could be to leave the value unset in the migration
(therefore doing a DDL only migration), and then handle missing IP
addresses in application code.
But if these are "default" values which can change in the future
independently of the VPN's router GW IP, then I'd suggest doing that in the
migration following your approach.




>
>
>>
>>>
>>> Appreciate any advise here on how to debug the migration stuff...
>>>
>>> Paul Michali (pc_m)
>>>
>>
>> [1]
>> http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124
>>
>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150707/3f5ebe82/attachment.html>