<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On 7 July 2015 at 14:00, Paul Michali <span dir="ltr"><<a href="mailto:pc@michali.net" target="_blank">pc@michali.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Thanks Salvatore for the responses. See @PCM in-line...<div><br></div><div><br><br><div class="gmail_quote"><span class=""><div dir="ltr">On Tue, Jul 7, 2015 at 6:14 AM Salvatore Orlando <<a href="mailto:sorlando@nicira.com" target="_blank">sorlando@nicira.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Some comments inline.<div><br></div><div>Salvatore<br><div class="gmail_extra"><br><div class="gmail_quote"></div></div></div></div><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote">On 6 July 2015 at 20:00, Paul Michali <span dir="ltr"><<a href="mailto:pc@michali.net" target="_blank">pc@michali.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>I have some urgent requests about migration that I'm hoping to get some info on. I'm working on a bug where I need to add two (related) fields to a table for VPNaaS. Here's the objectives related to migration...</div><div><br></div><div>1) create local_v4_ip and lcoal_v6_ip fields in the vpnservice table</div><div>2) for each entry in the vpnservice table:</div><div> 2.1) Get the router.gw_port.fixed_ips list</div><div> 2.2) Determine the version of each fixed IP and store the first of each version (if any) into the appropriate new field.</div><div><br></div><div>I have created a migration file, and I changed the down_revision to be the number of the revision that is the first in the migration chain in the VPN repo.</div><div><br></div><div>Here are the many questions I have...</div><div><br></div><div>When I look in the VPN repo, the HEAD file has the version 'kilo', which is not the current head. </div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>Shouldn't it the version number of the first file in the migration chain?</div></div></blockquote><div><br></div></div></div></div></div><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div>It should indeed. How are you generating the revision script? Using neutron-db-manage it should be updated automatically [1]</div></div></div></div></div></blockquote><div><br></div></span><div>@PCM I ran neutron-db-manage, when in the neutron repo, and it assigned some version, but it was not the latest in the neutron-vpnaas repo.</div></div></div></div></blockquote><div><br></div><div>when you create a revision Alembic automatically assigns it a unique id. However, the neutron migration CLI (neutron-db-manage) then should take care of updating the HEAD file automatically. If this is not happening, that's where the problem should be.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div class="gmail_quote"><div><br></div><div>I checked the VPN repo and there were a chain of versions, which I used to determine what the head should be and have set the version accordingly. However, in the current repo, head is set to "kilo", which appears to be incorrect. The versions are:<br></div><div><br></div><div>56893333aa52</div><div>kilo <<< HEAD</div><div>3ea02b2a773e</div><div>start_neutron_vpnaas</div><div>None</div><div><br></div><div>Should I do a separate commit that fixes the HEAD file, or just fix it as part of the bug fix I'm working on.</div></div></div></div></blockquote><div><br></div><div>In order to pass functional tests the HEAD file must point to the topmost revision (56893333aa52)</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div class="gmail_quote"><div><br></div><div>BTW, at one point, after having correctly set the HEAD and versions in my new migration file, I think I ran neutron-db-manage check_migration, and I think it set the HEAD to my version, but it did that in the neutron repo, and not the VPN repo. I might have been running from the wrong repo?</div></div></div></div></blockquote><div><br></div><div>Yes, probably.</div><div>neutron-db-manage by default works on the neutron repo. In order to work with a service repo you should specify it on the command line (<a href="http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n41">http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n41</a>).</div><div>This might also explain why the HEAD is not getting updated in your repo.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div class="gmail_quote"><span class=""><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>For my commit, I'm assuming I change the HEAD file to use my migration file's version?</div></div></blockquote><div><br></div></div></div></div></div><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div>You can do that manually too, yes.</div></div></div></div></div><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>I set HEAD to my migration file, and my file has a down revision of the previous head's revision. If I run 'neutron-db-manage --config-file ../neutron/etc/neutron.conf --config-file ../neutron/etc/neutron/plugins/ml2/ml2_conf.ini check_migration' there is no output so I guess that is OK.</div><div><br></div><div>As I develop my new migration file, is there a way that I can test it (running neutron-db-migration, maybe)?</div></div></blockquote><div><br></div></div></div></div></div><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div>When I test migrations I usually dump the database, run the migration with neutron-db-manage upgrade HEAD (I think it's not necessary to specify HEAD), and restore the db from the dump if the migration fails.</div></div></div></div></div><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>Is there a way to run the migration file under the debugger, as well (importing pdb, for example)?</div></div></blockquote><div><br></div></div></div></div></div><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div>The migration process is just like any python application, so I guess you can debug it with pdb. </div></div></div></div></div></blockquote><div><br></div></span><div>@PCM Ah, so use "neutron-db-manage upgrade HEAD". That was the piece that was missing. I take it there are no specific unit tests of the migration files?</div></div></div></div></blockquote><div><br></div><div>... and also specify --service vpnaas </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div class="gmail_quote"><span class=""><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>In the migration, I can add the columns needed. What's the best way to fill out those fields - using raw SQL queries or create a Session object and access the VpnService object's router object?</div></div></blockquote><div><br></div></div></div></div></div><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div>If the default value for the column is not enough, and you need to specify a value which depends on other values in the same row I would prefer plain SQL statements, but if that become cumbersome I guess it's ok to use sqlalchemy's session.</div></div></div></div></div><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>I see there is some op.bind() call and then engine.execute(), but could use some help on the best way to extract the needed queries (I need to access the vpnservice's router, and then access the (Port) gw_port relationship, and from that access the (IPAllocation) fixed_ips list).</div></div></blockquote><div><br></div></div></div></div></div><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div>Perhaps you can point us to the review pages on gerrit, and we can provide detailed comments there.</div></div></div></div></div></blockquote><div><br></div></span><div>@PCM Yeah, I haven't pushed it up yet. I have a few more changes to make, and should be able to get it up in a few days. The LP bug is 1464387.</div><div><br></div><div>Essentially, in the vpnservices table, I'm adding an IPv4 and/or IPv6 addresses for the "local" end of VPN connections that will be established. This is to allow alternative VPN implementation (appliances, separate S/W, H/W, VM based VPN, etc) to specify addresses different than what is available on the Neutron router.</div><div><br></div><div>However, for the reference implementation, we'll use the Neutron router's fixed_ips list (as is done today), and to handle the migration, I'm thinking the following is needed:</div><div><br></div><div>1) create the new columns.</div><div>2) Identify the router for that service and obtain it's GW fixed_ips list.</div><div>3) Pick first IPv4 address (if any) and IPv6 address (if any), and store in new columns.</div><div><br></div><div>So I need to form a query and code to do this.</div><span class=""><div><br></div></span></div></div></div></blockquote><div><br></div><div>You can put all this logic in the DB migration and it should work, from what I can tell.</div><div>An alternative could be to leave the value unset in the migration (therefore doing a DDL only migration), and then handle missing IP addresses in application code.</div><div>But if these are "default" values which can change in the future independently of the VPN's router GW IP, then I'd suggest doing that in the migration following your approach.</div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div class="gmail_quote"><span class=""><div></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>Appreciate any advise here on how to debug the migration stuff...</div><div><br></div><div>Paul Michali (pc_m)</div></div></blockquote><div><br></div></div></div></div></div><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div>[1] <a href="http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124" target="_blank">http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124</a> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div></div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></span></div></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div>