<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On Tue, Jul 07, 2015, Salvatore Orlando
<a class="moz-txt-link-rfc2396E" href="mailto:sorlando@nicira.com"><sorlando@nicira.com></a> wrote:<br>
</div>
<blockquote
cite="mid:CAGR=i3hcna9HAyLd4k0P8xu4U3jyZTqRvZWwdoyfVH4=gR6ruA@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 7 July 2015 at 14:00, Paul Michali
<span dir="ltr"><<a moz-do-not-send="true"
href="mailto:pc@michali.net" target="_blank">pc@michali.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Thanks Salvatore for the responses. See
@PCM in-line...
<div><br>
</div>
<div><br>
<br>
<div class="gmail_quote"><span class="">
<div dir="ltr">On Tue, Jul 7, 2015 at 6:14 AM
Salvatore Orlando <<a moz-do-not-send="true"
href="mailto:sorlando@nicira.com"
target="_blank">sorlando@nicira.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Some comments inline.
<div><br>
</div>
<div>Salvatore<br>
<div class="gmail_extra"><br>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">On 6 July 2015 at
20:00, Paul Michali <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:pc@michali.net"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:pc@michali.net">pc@michali.net</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Hi,
<div><br>
</div>
<div>I have some urgent requests
about migration that I'm hoping to
get some info on. I'm working on a
bug where I need to add two
(related) fields to a table for
VPNaaS. Here's the objectives
related to migration...</div>
<div><br>
</div>
<div>1) create local_v4_ip and
lcoal_v6_ip fields in the
vpnservice table</div>
<div>2) for each entry in the
vpnservice table:</div>
<div> 2.1) Get the
router.gw_port.fixed_ips list</div>
<div> 2.2) Determine the version
of each fixed IP and store the
first of each version (if any)
into the appropriate new field.</div>
<div><br>
</div>
<div>I have created a migration
file, and I changed the
down_revision to be the number of
the revision that is the first in
the migration chain in the VPN
repo.</div>
<div><br>
</div>
<div>Here are the many questions I
have...</div>
<div><br>
</div>
<div>When I look in the VPN repo,
the HEAD file has the version
'kilo', which is not the current
head. </div>
</div>
</blockquote>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
<div>Shouldn't it the version number
of the first file in the migration
chain?</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>It should indeed. How are you
generating the revision script? Using
neutron-db-manage it should be updated
automatically [1]</div>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>@PCM I ran neutron-db-manage, when in the
neutron repo, and it assigned some version, but it
was not the latest in the neutron-vpnaas repo.</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>when you create a revision Alembic automatically
assigns it a unique id. However, the neutron migration CLI
(neutron-db-manage) then should take care of updating the
HEAD file automatically. If this is not happening, that's
where the problem should be.</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>
<div class="gmail_quote">
<div><br>
</div>
<div>I checked the VPN repo and there were a chain
of versions, which I used to determine what the
head should be and have set the version
accordingly. However, in the current repo, head is
set to "kilo", which appears to be incorrect. The
versions are:<br>
</div>
<div><br>
</div>
<div>56893333aa52</div>
<div>kilo <<< HEAD</div>
<div>3ea02b2a773e</div>
<div>start_neutron_vpnaas</div>
<div>None</div>
<div><br>
</div>
<div>Should I do a separate commit that fixes the
HEAD file, or just fix it as part of the bug fix
I'm working on.</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>In order to pass functional tests the HEAD file must
point to the topmost revision (56893333aa52)</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>
<div class="gmail_quote">
<div><br>
</div>
<div>BTW, at one point, after having correctly set
the HEAD and versions in my new migration file, I
think I ran neutron-db-manage check_migration, and
I think it set the HEAD to my version, but it did
that in the neutron repo, and not the VPN repo. I
might have been running from the wrong repo?</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Yes, probably.</div>
<div>neutron-db-manage by default works on the neutron repo.
In order to work with a service repo you should specify it
on the command line (<a moz-do-not-send="true"
href="http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n41">http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n41</a>).</div>
<div>This might also explain why the HEAD is not getting
updated in your repo.</div>
</div>
</div>
</div>
</blockquote>
I have been playing around with this. Unfortunately, --service only
works for the "upgrade" command. It does not work for the "revision"
command, and even less for --autogenerate.<br>
<br>
I am working on a fix.<br>
<br>
<blockquote
cite="mid:CAGR=i3hcna9HAyLd4k0P8xu4U3jyZTqRvZWwdoyfVH4=gR6ruA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>
<div class="gmail_quote"><span class="">
<div><br>
</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>For my commit, I'm assuming I
change the HEAD file to use my
migration file's version?</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>You can do that manually too, yes.</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
<div>I set HEAD to my migration
file, and my file has a down
revision of the previous head's
revision. If I run
'neutron-db-manage --config-file
../neutron/etc/neutron.conf
--config-file
../neutron/etc/neutron/plugins/ml2/ml2_conf.ini
check_migration' there is no
output so I guess that is OK.</div>
<div><br>
</div>
<div>As I develop my new migration
file, is there a way that I can
test it (running
neutron-db-migration, maybe)?</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>When I test migrations I usually
dump the database, run the migration
with neutron-db-manage upgrade HEAD (I
think it's not necessary to specify
HEAD), and restore the db from the
dump if the migration fails.</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div> <br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>Is there a way to run the
migration file under the debugger,
as well (importing pdb, for
example)?</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>The migration process is just like
any python application, so I guess you
can debug it with pdb. </div>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>@PCM Ah, so use "neutron-db-manage upgrade
HEAD". That was the piece that was missing. I take
it there are no specific unit tests of the
migration files?</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>... and also specify --service vpnaas </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>
<div class="gmail_quote"><span class="">
<div><br>
</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
<div>In the migration, I can add the
columns needed. What's the best
way to fill out those fields -
using raw SQL queries or create a
Session object and access the
VpnService object's router object?</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>If the default value for the column
is not enough, and you need to specify
a value which depends on other values
in the same row I would prefer plain
SQL statements, but if that become
cumbersome I guess it's ok to use
sqlalchemy's session.</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
<div>I see there is some op.bind()
call and then engine.execute(),
but could use some help on the
best way to extract the needed
queries (I need to access the
vpnservice's router, and then
access the (Port) gw_port
relationship, and from that access
the (IPAllocation) fixed_ips
list).</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>Perhaps you can point us to the
review pages on gerrit, and we can
provide detailed comments there.</div>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>@PCM Yeah, I haven't pushed it up yet. I have a
few more changes to make, and should be able to
get it up in a few days. The LP bug is 1464387.</div>
<div><br>
</div>
<div>Essentially, in the vpnservices table, I'm
adding an IPv4 and/or IPv6 addresses for the
"local" end of VPN connections that will be
established. This is to allow alternative VPN
implementation (appliances, separate S/W, H/W, VM
based VPN, etc) to specify addresses different
than what is available on the Neutron router.</div>
<div><br>
</div>
<div>However, for the reference implementation,
we'll use the Neutron router's fixed_ips list (as
is done today), and to handle the migration, I'm
thinking the following is needed:</div>
<div><br>
</div>
<div>1) create the new columns.</div>
<div>2) Identify the router for that service and
obtain it's GW fixed_ips list.</div>
<div>3) Pick first IPv4 address (if any) and IPv6
address (if any), and store in new columns.</div>
<div><br>
</div>
<div>So I need to form a query and code to do this.</div>
<span class="">
<div><br>
</div>
</span></div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>You can put all this logic in the DB migration and it
should work, from what I can tell.</div>
<div>An alternative could be to leave the value unset in the
migration (therefore doing a DDL only migration), and then
handle missing IP addresses in application code.</div>
<div>But if these are "default" values which can change in
the future independently of the VPN's router GW IP, then
I'd suggest doing that in the migration following your
approach.</div>
<div><br>
</div>
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>
<div class="gmail_quote"><span class="">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
<div>Appreciate any advise here on
how to debug the migration
stuff...</div>
<div><br>
</div>
<div>Paul Michali (pc_m)</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>[1] <a moz-do-not-send="true"
href="http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124"
target="_blank">http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124</a> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</span></div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>