<div dir="ltr">Salvatore,<div><br></div><div>I changed head to the version before my new one, and then tried to upgrade and I see this:</div><div><div> neutron-db-manage --config-file /opt/stack/neutron/etc/neutron.conf --service vpnaas upgrade HEAD</div><div>Traceback (most recent call last):</div><div>  File "/usr/local/bin/neutron-db-manage", line 10, in <module></div><div>    sys.exit(main())</div><div>  File "/opt/stack/neutron/neutron/db/migration/cli.py", line 238, in main</div><div>    CONF.command.func(config, <a href="http://CONF.command.name">CONF.command.name</a>)</div><div>  File "/opt/stack/neutron/neutron/db/migration/cli.py", line 105, in do_upgrade</div><div>    run_sanity_checks(config, revision)</div><div>  File "/opt/stack/neutron/neutron/db/migration/cli.py", line 229, in run_sanity_checks</div><div>    script_dir.run_env()</div><div>  File "/usr/local/lib/python2.7/dist-packages/alembic/script.py", line 390, in run_env</div><div>    util.load_python_file(self.dir, 'env.py')</div><div>  File "/usr/local/lib/python2.7/dist-packages/alembic/util.py", line 243, in load_python_file</div><div>    module = load_module_py(module_id, path)</div><div>  File "/usr/local/lib/python2.7/dist-packages/alembic/compat.py", line 79, in load_module_py</div><div>    mod = imp.load_source(module_id, path, fp)</div><div>  File "/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py", line 86, in <module></div><div>    run_migrations_online()</div><div>  File "/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py", line 67, in run_migrations_online</div><div>    engine = session.create_engine(neutron_config.database.connection)</div><div>  File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 112, in create_engine</div><div>    url = sqlalchemy.engine.url.make_url(sql_connection)</div><div>  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py", line 186, in make_url</div><div>    return _parse_rfc1738_args(name_or_url)</div><div>  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py", line 235, in _parse_rfc1738_args</div><div>    "Could not parse rfc1738 URL from string '%s'" % name)</div><div>sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string ''</div></div><div><br></div><div>Any ideas what is wrong here?</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jul 7, 2015 at 10:05 AM Paul Michali <<a href="mailto:pc@michali.net">pc@michali.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>Yes, I wasn't using the --service option, so I suspect that is why my down_version was wrong.  In talking with Akihiro, I added a check to PEP8 and made sure that it fails if head is wrong. It is: <a href="https://review.openstack.org/#/c/199082/" target="_blank">https://review.openstack.org/#/c/199082/</a> (of course that failed py27 - I've got to see if there was some recent breakage in vpn repo, again).</div><div><br></div><div>Regarding the migration, one of the new columns may be None, but there must be at least one IP version entry (there is an existing test in VPN for using a router w/o an external IP set). Since the new code will rely on these new fields, I'd like to populate them as part of the migration. I think it would be more complicated to handle during operation.</div><div><br></div><div>Does anyone have examples of how to do queries of objects, from the migration upgrade() code?</div><div><br></div><div><br></div><div>Regards,</div><div><br></div><div>PCM</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jul 7, 2015 at 9:02 AM Akihiro Motoki <<a href="mailto:amotoki@gmail.com" target="_blank">amotoki@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">2015-07-07 21:39 GMT+09:00 Henry Gessau <span dir="ltr"><<a href="mailto:gessau@cisco.com" target="_blank">gessau@cisco.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF" text="#000000"><span>
    <div>On Tue, Jul 07, 2015, Paul Michali
      <a href="mailto:pc@michali.net" target="_blank"><pc@michali.net></a> wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">Thanks Salvatore for the responses. See @PCM
        in-line...
        <div><br>
        </div>
        <div><br>
          <br>
          <div class="gmail_quote">
            <div dir="ltr">On Tue, Jul 7, 2015 at 6:14 AM Salvatore
              Orlando <<a href="mailto:sorlando@nicira.com" target="_blank">sorlando@nicira.com</a>>
              wrote:<br>
            </div>
            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div dir="ltr">Some comments inline.
                <div><br>
                </div>
                <div>Salvatore<br>
                  <div class="gmail_extra"><br>
                  </div>
                </div>
              </div>
              <div dir="ltr">
                <div>
                  <div class="gmail_extra">
                    <div class="gmail_quote">On 6 July 2015 at 20:00,
                      Paul Michali <span dir="ltr"><<a href="mailto:pc@michali.net" target="_blank"></a><a href="mailto:pc@michali.net" target="_blank">pc@michali.net</a>></span>
                      wrote:<br>
                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                        <div dir="ltr">Hi,
                          <div><br>
                          </div>
                          <div>I have some urgent requests about
                            migration that I'm hoping to get some info
                            on. I'm working on a bug where I need to add
                            two (related) fields to a table for VPNaaS.
                            Here's the objectives related to
                            migration...</div>
                          <div><br>
                          </div>
                          <div>1) create local_v4_ip and lcoal_v6_ip
                            fields in the vpnservice table</div>
                          <div>2) for each entry in the vpnservice
                            table:</div>
                          <div>    2.1) Get the router.gw_port.fixed_ips
                            list</div>
                          <div>    2.2) Determine the version of each
                            fixed IP and store the first of each version
                            (if any) into the appropriate new field.</div>
                          <div><br>
                          </div>
                          <div>I have created a migration file, and I
                            changed the down_revision to be the number
                            of the revision that is the first in the
                            migration chain in the VPN repo.</div>
                          <div><br>
                          </div>
                          <div>Here are the many questions I have...</div>
                          <div><br>
                          </div>
                          <div>When I look in the VPN repo, the HEAD
                            file has the version 'kilo', which is not
                            the current head. </div>
                        </div>
                      </blockquote>
                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                        <div dir="ltr">
                          <div><br>
                          </div>
                          <div>Shouldn't it the version number of the
                            first file in the migration chain?</div>
                        </div>
                      </blockquote>
                      <div><br>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
              <div dir="ltr">
                <div>
                  <div class="gmail_extra">
                    <div class="gmail_quote">
                      <div>It should indeed. How are you generating the
                        revision script? Using neutron-db-manage it
                        should be updated automatically [1]</div>
                    </div>
                  </div>
                </div>
              </div>
            </blockquote>
            <div><br>
            </div>
            <div>@PCM I ran neutron-db-manage, when in the neutron repo,
              and it assigned some version, but it was not the latest in
              the neutron-vpnaas repo.</div>
          </div>
        </div>
      </div>
    </blockquote></span>
    neutron-db-manage does not handle alembic branches in separate repos
    very well at all yet. I am working on updating it with <a href="https://review.openstack.org/198524" target="_blank"></a><a href="https://review.openstack.org/198524" target="_blank">https://review.openstack.org/198524</a>
    but I have quite a lot left to do.</div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Yes, at now we have implicit order of running alembic migrations.</div><div>First run neutron db migration and then advanced service migrations.</div><div><br></div><div>I do not fully understand how alembic branch mechanism works, but</div><div>I think we can have a common ancestor and have multiple branches,</div><div>and each branch can evolve independently.</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><span><blockquote type="cite"><div dir="ltr"><div><div class="gmail_quote"><div><br>
            </div>
            <div>I checked the VPN repo and there were a chain of
              versions, which I used to determine what the head should
              be and have set the version accordingly. However, in the
              current repo, head is set to "kilo", which appears to be
              incorrect.  The versions are:<br>
            </div>
            <div><br>
            </div>
            <div>56893333aa52</div>
            <div>kilo   <<< HEAD</div>
            <div>3ea02b2a773e</div>
            <div>start_neutron_vpnaas</div>
            <div>None</div>
          </div>
        </div>
      </div>
    </blockquote></span>
    Ouch. That is an error, because <a href="https://review.openstack.org/190569" target="_blank"></a><a href="https://review.openstack.org/190569" target="_blank">https://review.openstack.org/190569</a>
    should have updated HEAD but didn't.<br>
    <br>
    The version sequence (you can see it in any devstack run) is:<br>
    <br>
    INFO  [alembic.migration] Running upgrade  ->
    start_neutron_vpnaas, start neutron-vpnaas chain<br>
    INFO  [alembic.migration] Running upgrade start_neutron_vpnaas ->
    3ea02b2a773e, add_index_tenant_id<br>
    INFO  [alembic.migration] Running upgrade 3ea02b2a773e -> kilo,
    kilo<br>
    INFO  [alembic.migration] Running upgrade kilo -> 56893333aa52,
    fix identifier map fk</div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>It seems we don't have an appropriate check for HEAD revision in at least VPNaaS repo.</div><div>Paul and I just discussed it. We need to improve the check too.</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><span><blockquote type="cite"><div dir="ltr"><div><div class="gmail_quote">
            <div>Should I do a separate commit that fixes the HEAD file,
              or just fix it as part of the bug fix I'm working on.</div>
          </div>
        </div>
      </div>
    </blockquote></span>
    Yes, you should immediately submit a patch to change HEAD to
    56893333aa52.<span><br>
    <br>
    <blockquote type="cite">
      <div dir="ltr">
        <div>
          <div class="gmail_quote">
            <div><br>
            </div>
            <div>BTW, at one point, after having correctly set the HEAD
              and versions in my new migration file, I think I ran
              neutron-db-manage check_migration, and I think it set the
              HEAD to my version, but it did that in the neutron repo,
              and not the VPN repo.  I might have been running from the
              wrong repo?</div>
          </div>
        </div>
      </div>
    </blockquote></span>
    I working on updating the devref docs for this process. Things have
    changed quite a bit with the alembic branches in separate repos.<span><br>
    <br>
    <blockquote type="cite">
      <div dir="ltr">
        <div>
          <div class="gmail_quote">
            <div><br>
            </div>
            <div><br>
            </div>
            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div dir="ltr">
                <div>
                  <div class="gmail_extra">
                    <div class="gmail_quote">
                      <div><br>
                      </div>
                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                        <div dir="ltr">
                          <div>For my commit, I'm assuming I change the
                            HEAD file to use my migration file's
                            version?</div>
                        </div>
                      </blockquote>
                      <div><br>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
              <div dir="ltr">
                <div>
                  <div class="gmail_extra">
                    <div class="gmail_quote">
                      <div>You can do that manually too, yes.</div>
                    </div>
                  </div>
                </div>
              </div>
              <div dir="ltr">
                <div>
                  <div class="gmail_extra">
                    <div class="gmail_quote">
                      <div> </div>
                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                        <div dir="ltr">
                          <div><br>
                          </div>
                          <div>I set HEAD to my migration file, and my
                            file has a down revision of the previous
                            head's revision. If I run 'neutron-db-manage
                            --config-file ../neutron/etc/neutron.conf
                            --config-file
                            ../neutron/etc/neutron/plugins/ml2/ml2_conf.ini
                            check_migration' there is no output so I
                            guess that is OK.</div>
                          <div><br>
                          </div>
                          <div>As I develop my new migration file, is
                            there a way that I can test it (running
                            neutron-db-migration, maybe)?</div>
                        </div>
                      </blockquote>
                      <div><br>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
              <div dir="ltr">
                <div>
                  <div class="gmail_extra">
                    <div class="gmail_quote">
                      <div>When I test migrations I usually dump the
                        database, run the migration with
                        neutron-db-manage upgrade HEAD (I think it's not
                        necessary to specify HEAD), and restore the db
                        from the dump if the migration fails.</div>
                    </div>
                  </div>
                </div>
              </div>
              <div dir="ltr">
                <div>
                  <div class="gmail_extra">
                    <div class="gmail_quote">
                      <div> <br>
                      </div>
                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                        <div dir="ltr">
                          <div>Is there a way to run the migration file
                            under the debugger, as well (importing pdb,
                            for example)?</div>
                        </div>
                      </blockquote>
                      <div><br>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
              <div dir="ltr">
                <div>
                  <div class="gmail_extra">
                    <div class="gmail_quote">
                      <div>The migration process is just like any python
                        application, so I guess you can debug it with
                        pdb. </div>
                    </div>
                  </div>
                </div>
              </div>
            </blockquote>
            <div><br>
            </div>
            <div>@PCM Ah, so use "neutron-db-manage upgrade HEAD". That
              was the piece that was missing. I take it there are no
              specific unit tests of the migration files?</div>
            <div><br>
            </div>
            <div><br>
            </div>
            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div dir="ltr">
                <div>
                  <div class="gmail_extra">
                    <div class="gmail_quote">
                      <div> </div>
                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                        <div dir="ltr">
                          <div><br>
                          </div>
                          <div>In the migration, I can add the columns
                            needed. What's the best way to fill out
                            those fields - using raw SQL queries or
                            create a Session object and access the
                            VpnService object's router object?</div>
                        </div>
                      </blockquote>
                      <div><br>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
              <div dir="ltr">
                <div>
                  <div class="gmail_extra">
                    <div class="gmail_quote">
                      <div>If the default value for the column is not
                        enough, and you need to specify a value which
                        depends on other values in the same row I would
                        prefer plain SQL statements, but if that become
                        cumbersome I guess it's ok to use sqlalchemy's
                        session.</div>
                    </div>
                  </div>
                </div>
              </div>
              <div dir="ltr">
                <div>
                  <div class="gmail_extra">
                    <div class="gmail_quote">
                      <div><br>
                      </div>
                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                        <div dir="ltr">
                          <div><br>
                          </div>
                          <div>I see there is some op.bind() call and
                            then engine.execute(), but could use some
                            help on the best way to extract the needed
                            queries (I need to access the vpnservice's
                            router, and then access the (Port) gw_port
                            relationship, and from that access the
                            (IPAllocation) fixed_ips list).</div>
                        </div>
                      </blockquote>
                      <div><br>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
              <div dir="ltr">
                <div>
                  <div class="gmail_extra">
                    <div class="gmail_quote">
                      <div>Perhaps you can point us to the review pages
                        on gerrit, and we can provide detailed comments
                        there.</div>
                    </div>
                  </div>
                </div>
              </div>
            </blockquote>
            <div><br>
            </div>
            <div>@PCM Yeah, I haven't pushed it up yet. I have a few
              more changes to make, and should be able to get it up in a
              few days. The LP bug is 1464387.</div>
            <div><br>
            </div>
            <div>Essentially, in the vpnservices table, I'm adding an
              IPv4 and/or IPv6 addresses for the  "local" end of VPN
              connections that will be established. This is to allow
              alternative VPN implementation (appliances, separate S/W,
              H/W, VM based VPN, etc) to specify addresses different
              than what is available on the Neutron router.</div>
            <div><br>
            </div>
            <div>However, for the reference implementation, we'll use
              the Neutron router's fixed_ips list (as is done today),
              and to handle the migration, I'm thinking the following is
              needed:</div>
            <div><br>
            </div>
            <div>1) create the new columns.</div>
            <div>2) Identify the router for that service and obtain it's
              GW fixed_ips list.</div>
            <div>3) Pick first IPv4 address (if any) and IPv6 address
              (if any), and store in new columns.</div>
            <div><br>
            </div>
            <div>So I need to form a query and code to do this.</div>
            <div><br>
            </div>
            <div><br>
            </div>
            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div dir="ltr">
                <div>
                  <div class="gmail_extra">
                    <div class="gmail_quote">
                      <div> </div>
                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                        <div dir="ltr">
                          <div><br>
                          </div>
                          <div>Appreciate any advise here on how to
                            debug the migration stuff...</div>
                          <div><br>
                          </div>
                          <div>Paul Michali (pc_m)</div>
                        </div>
                      </blockquote>
                      <div><br>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
              <div dir="ltr">
                <div>
                  <div class="gmail_extra">
                    <div class="gmail_quote">
                      <div>[1] <a href="http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124" target="_blank">http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124</a> </div>
                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                        <div dir="ltr">
                          <div><br>
                          </div>
                          <br>
                        </div>
                      </blockquote>
                    </div>
                  </div>
                </div>
              </div>
            </blockquote>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
  </span></div>

<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div></div></div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div></blockquote></div>