<div dir="ltr">possibly I was wrong in mixing up git & alembic.<div>It should be "upgrade head" - lowercase.</div><div><br></div><div>If that doesn't work there might some other issue lurking.</div><div><br></div><div>Salvatore</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 7 July 2015 at 17:44, Paul Michali <span dir="ltr"><<a href="mailto:pc@michali.net" target="_blank">pc@michali.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Salvatore,<div><br></div><div>I changed head to the version before my new one, and then tried to upgrade and I see this:</div><div><div> neutron-db-manage --config-file /opt/stack/neutron/etc/neutron.conf --service vpnaas upgrade HEAD</div><div>Traceback (most recent call last):</div><div> File "/usr/local/bin/neutron-db-manage", line 10, in <module></div><div> sys.exit(main())</div><div> File "/opt/stack/neutron/neutron/db/migration/cli.py", line 238, in main</div><div> CONF.command.func(config, <a href="http://CONF.command.name" target="_blank">CONF.command.name</a>)</div><div> File "/opt/stack/neutron/neutron/db/migration/cli.py", line 105, in do_upgrade</div><div> run_sanity_checks(config, revision)</div><div> File "/opt/stack/neutron/neutron/db/migration/cli.py", line 229, in run_sanity_checks</div><div> script_dir.run_env()</div><div> File "/usr/local/lib/python2.7/dist-packages/alembic/script.py", line 390, in run_env</div><div> util.load_python_file(self.dir, 'env.py')</div><div> File "/usr/local/lib/python2.7/dist-packages/alembic/util.py", line 243, in load_python_file</div><div> module = load_module_py(module_id, path)</div><div> File "/usr/local/lib/python2.7/dist-packages/alembic/compat.py", line 79, in load_module_py</div><div> mod = imp.load_source(module_id, path, fp)</div><div> File "/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py", line 86, in <module></div><div> run_migrations_online()</div><div> File "/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py", line 67, in run_migrations_online</div><div> engine = session.create_engine(neutron_config.database.connection)</div><div> File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 112, in create_engine</div><div> url = sqlalchemy.engine.url.make_url(sql_connection)</div><div> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py", line 186, in make_url</div><div> return _parse_rfc1738_args(name_or_url)</div><div> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py", line 235, in _parse_rfc1738_args</div><div> "Could not parse rfc1738 URL from string '%s'" % name)</div><div>sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string ''</div></div><div><br></div><div>Any ideas what is wrong here?</div></div><div class="HOEnZb"><div class="h5"><br><div class="gmail_quote"><div dir="ltr">On Tue, Jul 7, 2015 at 10:05 AM Paul Michali <<a href="mailto:pc@michali.net" target="_blank">pc@michali.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>Yes, I wasn't using the --service option, so I suspect that is why my down_version was wrong. In talking with Akihiro, I added a check to PEP8 and made sure that it fails if head is wrong. It is: <a href="https://review.openstack.org/#/c/199082/" target="_blank">https://review.openstack.org/#/c/199082/</a> (of course that failed py27 - I've got to see if there was some recent breakage in vpn repo, again).</div><div><br></div><div>Regarding the migration, one of the new columns may be None, but there must be at least one IP version entry (there is an existing test in VPN for using a router w/o an external IP set). Since the new code will rely on these new fields, I'd like to populate them as part of the migration. I think it would be more complicated to handle during operation.</div><div><br></div><div>Does anyone have examples of how to do queries of objects, from the migration upgrade() code?</div><div><br></div><div><br></div><div>Regards,</div><div><br></div><div>PCM</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jul 7, 2015 at 9:02 AM Akihiro Motoki <<a href="mailto:amotoki@gmail.com" target="_blank">amotoki@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">2015-07-07 21:39 GMT+09:00 Henry Gessau <span dir="ltr"><<a href="mailto:gessau@cisco.com" target="_blank">gessau@cisco.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><span>
<div>On Tue, Jul 07, 2015, Paul Michali
<a href="mailto:pc@michali.net" target="_blank"><pc@michali.net></a> wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Thanks Salvatore for the responses. See @PCM
in-line...
<div><br>
</div>
<div><br>
<br>
<div class="gmail_quote">
<div dir="ltr">On Tue, Jul 7, 2015 at 6:14 AM Salvatore
Orlando <<a href="mailto:sorlando@nicira.com" target="_blank">sorlando@nicira.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Some comments inline.
<div><br>
</div>
<div>Salvatore<br>
<div class="gmail_extra"><br>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">On 6 July 2015 at 20:00,
Paul Michali <span dir="ltr"><<a href="mailto:pc@michali.net" target="_blank"></a><a href="mailto:pc@michali.net" target="_blank">pc@michali.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Hi,
<div><br>
</div>
<div>I have some urgent requests about
migration that I'm hoping to get some info
on. I'm working on a bug where I need to add
two (related) fields to a table for VPNaaS.
Here's the objectives related to
migration...</div>
<div><br>
</div>
<div>1) create local_v4_ip and lcoal_v6_ip
fields in the vpnservice table</div>
<div>2) for each entry in the vpnservice
table:</div>
<div> 2.1) Get the router.gw_port.fixed_ips
list</div>
<div> 2.2) Determine the version of each
fixed IP and store the first of each version
(if any) into the appropriate new field.</div>
<div><br>
</div>
<div>I have created a migration file, and I
changed the down_revision to be the number
of the revision that is the first in the
migration chain in the VPN repo.</div>
<div><br>
</div>
<div>Here are the many questions I have...</div>
<div><br>
</div>
<div>When I look in the VPN repo, the HEAD
file has the version 'kilo', which is not
the current head. </div>
</div>
</blockquote>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
<div>Shouldn't it the version number of the
first file in the migration chain?</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>It should indeed. How are you generating the
revision script? Using neutron-db-manage it
should be updated automatically [1]</div>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>@PCM I ran neutron-db-manage, when in the neutron repo,
and it assigned some version, but it was not the latest in
the neutron-vpnaas repo.</div>
</div>
</div>
</div>
</blockquote></span>
neutron-db-manage does not handle alembic branches in separate repos
very well at all yet. I am working on updating it with <a href="https://review.openstack.org/198524" target="_blank"></a><a href="https://review.openstack.org/198524" target="_blank">https://review.openstack.org/198524</a>
but I have quite a lot left to do.</div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Yes, at now we have implicit order of running alembic migrations.</div><div>First run neutron db migration and then advanced service migrations.</div><div><br></div><div>I do not fully understand how alembic branch mechanism works, but</div><div>I think we can have a common ancestor and have multiple branches,</div><div>and each branch can evolve independently.</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><span><blockquote type="cite"><div dir="ltr"><div><div class="gmail_quote"><div><br>
</div>
<div>I checked the VPN repo and there were a chain of
versions, which I used to determine what the head should
be and have set the version accordingly. However, in the
current repo, head is set to "kilo", which appears to be
incorrect. The versions are:<br>
</div>
<div><br>
</div>
<div>56893333aa52</div>
<div>kilo <<< HEAD</div>
<div>3ea02b2a773e</div>
<div>start_neutron_vpnaas</div>
<div>None</div>
</div>
</div>
</div>
</blockquote></span>
Ouch. That is an error, because <a href="https://review.openstack.org/190569" target="_blank"></a><a href="https://review.openstack.org/190569" target="_blank">https://review.openstack.org/190569</a>
should have updated HEAD but didn't.<br>
<br>
The version sequence (you can see it in any devstack run) is:<br>
<br>
INFO [alembic.migration] Running upgrade ->
start_neutron_vpnaas, start neutron-vpnaas chain<br>
INFO [alembic.migration] Running upgrade start_neutron_vpnaas ->
3ea02b2a773e, add_index_tenant_id<br>
INFO [alembic.migration] Running upgrade 3ea02b2a773e -> kilo,
kilo<br>
INFO [alembic.migration] Running upgrade kilo -> 56893333aa52,
fix identifier map fk</div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>It seems we don't have an appropriate check for HEAD revision in at least VPNaaS repo.</div><div>Paul and I just discussed it. We need to improve the check too.</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><span><blockquote type="cite"><div dir="ltr"><div><div class="gmail_quote">
<div>Should I do a separate commit that fixes the HEAD file,
or just fix it as part of the bug fix I'm working on.</div>
</div>
</div>
</div>
</blockquote></span>
Yes, you should immediately submit a patch to change HEAD to
56893333aa52.<span><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div class="gmail_quote">
<div><br>
</div>
<div>BTW, at one point, after having correctly set the HEAD
and versions in my new migration file, I think I ran
neutron-db-manage check_migration, and I think it set the
HEAD to my version, but it did that in the neutron repo,
and not the VPN repo. I might have been running from the
wrong repo?</div>
</div>
</div>
</div>
</blockquote></span>
I working on updating the devref docs for this process. Things have
changed quite a bit with the alembic branches in separate repos.<span><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div class="gmail_quote">
<div><br>
</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>For my commit, I'm assuming I change the
HEAD file to use my migration file's
version?</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>You can do that manually too, yes.</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
<div>I set HEAD to my migration file, and my
file has a down revision of the previous
head's revision. If I run 'neutron-db-manage
--config-file ../neutron/etc/neutron.conf
--config-file
../neutron/etc/neutron/plugins/ml2/ml2_conf.ini
check_migration' there is no output so I
guess that is OK.</div>
<div><br>
</div>
<div>As I develop my new migration file, is
there a way that I can test it (running
neutron-db-migration, maybe)?</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>When I test migrations I usually dump the
database, run the migration with
neutron-db-manage upgrade HEAD (I think it's not
necessary to specify HEAD), and restore the db
from the dump if the migration fails.</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>Is there a way to run the migration file
under the debugger, as well (importing pdb,
for example)?</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>The migration process is just like any python
application, so I guess you can debug it with
pdb. </div>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>@PCM Ah, so use "neutron-db-manage upgrade HEAD". That
was the piece that was missing. I take it there are no
specific unit tests of the migration files?</div>
<div><br>
</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
<div>In the migration, I can add the columns
needed. What's the best way to fill out
those fields - using raw SQL queries or
create a Session object and access the
VpnService object's router object?</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>If the default value for the column is not
enough, and you need to specify a value which
depends on other values in the same row I would
prefer plain SQL statements, but if that become
cumbersome I guess it's ok to use sqlalchemy's
session.</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
<div>I see there is some op.bind() call and
then engine.execute(), but could use some
help on the best way to extract the needed
queries (I need to access the vpnservice's
router, and then access the (Port) gw_port
relationship, and from that access the
(IPAllocation) fixed_ips list).</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>Perhaps you can point us to the review pages
on gerrit, and we can provide detailed comments
there.</div>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>@PCM Yeah, I haven't pushed it up yet. I have a few
more changes to make, and should be able to get it up in a
few days. The LP bug is 1464387.</div>
<div><br>
</div>
<div>Essentially, in the vpnservices table, I'm adding an
IPv4 and/or IPv6 addresses for the "local" end of VPN
connections that will be established. This is to allow
alternative VPN implementation (appliances, separate S/W,
H/W, VM based VPN, etc) to specify addresses different
than what is available on the Neutron router.</div>
<div><br>
</div>
<div>However, for the reference implementation, we'll use
the Neutron router's fixed_ips list (as is done today),
and to handle the migration, I'm thinking the following is
needed:</div>
<div><br>
</div>
<div>1) create the new columns.</div>
<div>2) Identify the router for that service and obtain it's
GW fixed_ips list.</div>
<div>3) Pick first IPv4 address (if any) and IPv6 address
(if any), and store in new columns.</div>
<div><br>
</div>
<div>So I need to form a query and code to do this.</div>
<div><br>
</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
<div>Appreciate any advise here on how to
debug the migration stuff...</div>
<div><br>
</div>
<div>Paul Michali (pc_m)</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>[1] <a href="http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124" target="_blank">http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124</a> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
</span></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div></div></div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div></blockquote></div>
</div></div><br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>