Open Stack

On Tue, Jul 07, 2015, Salvatore Orlando <sorlando at nicira.com> wrote:
>
>
> On 7 July 2015 at 14:00, Paul Michali <pc at michali.net
> <mailto:pc at michali.net>> wrote:
>
>     Thanks Salvatore for the responses. See @PCM in-line...
>
>
>
>     On Tue, Jul 7, 2015 at 6:14 AM Salvatore Orlando <sorlando at nicira.com
>     <mailto:sorlando at nicira.com>> wrote:
>
>         Some comments inline.
>
>         Salvatore
>
>         On 6 July 2015 at 20:00, Paul Michali <pc at michali.net
>         <mailto:pc at michali.net>> wrote:
>
>             Hi,
>
>             I have some urgent requests about migration that I'm hoping to
>             get some info on. I'm working on a bug where I need to add two
>             (related) fields to a table for VPNaaS. Here's the objectives
>             related to migration...
>
>             1) create local_v4_ip and lcoal_v6_ip fields in the vpnservice table
>             2) for each entry in the vpnservice table:
>                 2.1) Get the router.gw_port.fixed_ips list
>                 2.2) Determine the version of each fixed IP and store the
>             first of each version (if any) into the appropriate new field.
>
>             I have created a migration file, and I changed the down_revision
>             to be the number of the revision that is the first in the
>             migration chain in the VPN repo.
>
>             Here are the many questions I have...
>
>             When I look in the VPN repo, the HEAD file has the version
>             'kilo', which is not the current head. 
>
>
>             Shouldn't it the version number of the first file in the
>             migration chain?
>
>
>         It should indeed. How are you generating the revision script? Using
>         neutron-db-manage it should be updated automatically [1]
>
>
>     @PCM I ran neutron-db-manage, when in the neutron repo, and it assigned
>     some version, but it was not the latest in the neutron-vpnaas repo.
>
>
> when you create a revision Alembic automatically assigns it a unique id.
> However, the neutron migration CLI (neutron-db-manage) then should take care
> of updating the HEAD file automatically. If this is not happening, that's
> where the problem should be.
>
>
>     I checked the VPN repo and there were a chain of versions, which I used
>     to determine what the head should be and have set the version
>     accordingly. However, in the current repo, head is set to "kilo", which
>     appears to be incorrect.  The versions are:
>
>     56893333aa52
>     kilo   <<< HEAD
>     3ea02b2a773e
>     start_neutron_vpnaas
>     None
>
>     Should I do a separate commit that fixes the HEAD file, or just fix it as
>     part of the bug fix I'm working on.
>
>
> In order to pass functional tests the HEAD file must point to the topmost
> revision (56893333aa52)
>
>
>     BTW, at one point, after having correctly set the HEAD and versions in my
>     new migration file, I think I ran neutron-db-manage check_migration, and
>     I think it set the HEAD to my version, but it did that in the neutron
>     repo, and not the VPN repo.  I might have been running from the wrong repo?
>
>
> Yes, probably.
> neutron-db-manage by default works on the neutron repo. In order to work with
> a service repo you should specify it on the command line
> (http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n41).
> This might also explain why the HEAD is not getting updated in your repo.
I have been playing around with this. Unfortunately, --service only works for
the "upgrade" command. It does not work for the "revision" command, and even
less for --autogenerate.

I am working on a fix.

>  
>
>
>
>
>             For my commit, I'm assuming I change the HEAD file to use my
>             migration file's version?
>
>
>         You can do that manually too, yes.
>          
>
>
>             I set HEAD to my migration file, and my file has a down revision
>             of the previous head's revision. If I run 'neutron-db-manage
>             --config-file ../neutron/etc/neutron.conf --config-file
>             ../neutron/etc/neutron/plugins/ml2/ml2_conf.ini check_migration'
>             there is no output so I guess that is OK.
>
>             As I develop my new migration file, is there a way that I can
>             test it (running neutron-db-migration, maybe)?
>
>
>         When I test migrations I usually dump the database, run the migration
>         with neutron-db-manage upgrade HEAD (I think it's not necessary to
>         specify HEAD), and restore the db from the dump if the migration fails.
>          
>
>             Is there a way to run the migration file under the debugger, as
>             well (importing pdb, for example)?
>
>
>         The migration process is just like any python application, so I guess
>         you can debug it with pdb. 
>
>
>     @PCM Ah, so use "neutron-db-manage upgrade HEAD". That was the piece that
>     was missing. I take it there are no specific unit tests of the migration
>     files?
>
>
> ... and also specify --service vpnaas 
>
>
>
>          
>
>
>             In the migration, I can add the columns needed. What's the best
>             way to fill out those fields - using raw SQL queries or create a
>             Session object and access the VpnService object's router object?
>
>
>         If the default value for the column is not enough, and you need to
>         specify a value which depends on other values in the same row I would
>         prefer plain SQL statements, but if that become cumbersome I guess
>         it's ok to use sqlalchemy's session.
>
>
>             I see there is some op.bind() call and then engine.execute(), but
>             could use some help on the best way to extract the needed queries
>             (I need to access the vpnservice's router, and then access the
>             (Port) gw_port relationship, and from that access the
>             (IPAllocation) fixed_ips list).
>
>
>         Perhaps you can point us to the review pages on gerrit, and we can
>         provide detailed comments there.
>
>
>     @PCM Yeah, I haven't pushed it up yet. I have a few more changes to make,
>     and should be able to get it up in a few days. The LP bug is 1464387.
>
>     Essentially, in the vpnservices table, I'm adding an IPv4 and/or IPv6
>     addresses for the  "local" end of VPN connections that will be
>     established. This is to allow alternative VPN implementation (appliances,
>     separate S/W, H/W, VM based VPN, etc) to specify addresses different than
>     what is available on the Neutron router.
>
>     However, for the reference implementation, we'll use the Neutron router's
>     fixed_ips list (as is done today), and to handle the migration, I'm
>     thinking the following is needed:
>
>     1) create the new columns.
>     2) Identify the router for that service and obtain it's GW fixed_ips list.
>     3) Pick first IPv4 address (if any) and IPv6 address (if any), and store
>     in new columns.
>
>     So I need to form a query and code to do this.
>
>
> You can put all this logic in the DB migration and it should work, from what
> I can tell.
> An alternative could be to leave the value unset in the migration (therefore
> doing a DDL only migration), and then handle missing IP addresses in
> application code.
> But if these are "default" values which can change in the future
> independently of the VPN's router GW IP, then I'd suggest doing that in the
> migration following your approach.
>
>
>  
>
>
>          
>
>
>             Appreciate any advise here on how to debug the migration stuff...
>
>             Paul Michali (pc_m)
>
>
>         [1]
>         http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124 
>
>
>
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150707/df4f022f/attachment.html>