Open Stack

Salvatore,

I changed head to the version before my new one, and then tried to upgrade
and I see this:
 neutron-db-manage --config-file /opt/stack/neutron/etc/neutron.conf
--service vpnaas upgrade HEAD
Traceback (most recent call last):
  File "/usr/local/bin/neutron-db-manage", line 10, in <module>
    sys.exit(main())
  File "/opt/stack/neutron/neutron/db/migration/cli.py", line 238, in main
    CONF.command.func(config, CONF.command.name)
  File "/opt/stack/neutron/neutron/db/migration/cli.py", line 105, in
do_upgrade
    run_sanity_checks(config, revision)
  File "/opt/stack/neutron/neutron/db/migration/cli.py", line 229, in
run_sanity_checks
    script_dir.run_env()
  File "/usr/local/lib/python2.7/dist-packages/alembic/script.py", line
390, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/local/lib/python2.7/dist-packages/alembic/util.py", line 243,
in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python2.7/dist-packages/alembic/compat.py", line 79,
in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File
"/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py",
line 86, in <module>
    run_migrations_online()
  File
"/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py",
line 67, in run_migrations_online
    engine = session.create_engine(neutron_config.database.connection)
  File
"/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py",
line 112, in create_engine
    url = sqlalchemy.engine.url.make_url(sql_connection)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py",
line 186, in make_url
    return _parse_rfc1738_args(name_or_url)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py",
line 235, in _parse_rfc1738_args
    "Could not parse rfc1738 URL from string '%s'" % name)
sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string ''

Any ideas what is wrong here?

On Tue, Jul 7, 2015 at 10:05 AM Paul Michali <pc at michali.net> wrote:

>
> Yes, I wasn't using the --service option, so I suspect that is why my
> down_version was wrong.  In talking with Akihiro, I added a check to PEP8
> and made sure that it fails if head is wrong. It is:
> https://review.openstack.org/#/c/199082/ (of course that failed py27 -
> I've got to see if there was some recent breakage in vpn repo, again).
>
> Regarding the migration, one of the new columns may be None, but there
> must be at least one IP version entry (there is an existing test in VPN for
> using a router w/o an external IP set). Since the new code will rely on
> these new fields, I'd like to populate them as part of the migration. I
> think it would be more complicated to handle during operation.
>
> Does anyone have examples of how to do queries of objects, from the
> migration upgrade() code?
>
>
> Regards,
>
> PCM
>
> On Tue, Jul 7, 2015 at 9:02 AM Akihiro Motoki <amotoki at gmail.com> wrote:
>
>> 2015-07-07 21:39 GMT+09:00 Henry Gessau <gessau at cisco.com>:
>>
>>>  On Tue, Jul 07, 2015, Paul Michali <pc at michali.net> <pc at michali.net>
>>> wrote:
>>>
>>> Thanks Salvatore for the responses. See @PCM in-line...
>>>
>>>
>>>
>>>  On Tue, Jul 7, 2015 at 6:14 AM Salvatore Orlando <sorlando at nicira.com>
>>> wrote:
>>>
>>>> Some comments inline.
>>>>
>>>>  Salvatore
>>>>
>>>>    On 6 July 2015 at 20:00, Paul Michali < <pc at michali.net>
>>>> pc at michali.net> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>>  I have some urgent requests about migration that I'm hoping to get
>>>>> some info on. I'm working on a bug where I need to add two (related) fields
>>>>> to a table for VPNaaS. Here's the objectives related to migration...
>>>>>
>>>>>  1) create local_v4_ip and lcoal_v6_ip fields in the vpnservice table
>>>>> 2) for each entry in the vpnservice table:
>>>>>     2.1) Get the router.gw_port.fixed_ips list
>>>>>     2.2) Determine the version of each fixed IP and store the first of
>>>>> each version (if any) into the appropriate new field.
>>>>>
>>>>>  I have created a migration file, and I changed the down_revision to
>>>>> be the number of the revision that is the first in the migration chain in
>>>>> the VPN repo.
>>>>>
>>>>>  Here are the many questions I have...
>>>>>
>>>>>  When I look in the VPN repo, the HEAD file has the version 'kilo',
>>>>> which is not the current head.
>>>>>
>>>>
>>>>>  Shouldn't it the version number of the first file in the migration
>>>>> chain?
>>>>>
>>>>
>>>>      It should indeed. How are you generating the revision script?
>>>> Using neutron-db-manage it should be updated automatically [1]
>>>>
>>>
>>>  @PCM I ran neutron-db-manage, when in the neutron repo, and it
>>> assigned some version, but it was not the latest in the neutron-vpnaas repo.
>>>
>>> neutron-db-manage does not handle alembic branches in separate repos
>>> very well at all yet. I am working on updating it with
>>> <https://review.openstack.org/198524>https://review.openstack.org/198524
>>> but I have quite a lot left to do.
>>>
>>
>> Yes, at now we have implicit order of running alembic migrations.
>> First run neutron db migration and then advanced service migrations.
>>
>> I do not fully understand how alembic branch mechanism works, but
>> I think we can have a common ancestor and have multiple branches,
>> and each branch can evolve independently.
>>
>>
>>>
>>>  I checked the VPN repo and there were a chain of versions, which I
>>> used to determine what the head should be and have set the version
>>> accordingly. However, in the current repo, head is set to "kilo", which
>>> appears to be incorrect.  The versions are:
>>>
>>>  56893333aa52
>>> kilo   <<< HEAD
>>> 3ea02b2a773e
>>> start_neutron_vpnaas
>>> None
>>>
>>> Ouch. That is an error, because <https://review.openstack.org/190569>
>>> https://review.openstack.org/190569 should have updated HEAD but didn't.
>>>
>>> The version sequence (you can see it in any devstack run) is:
>>>
>>> INFO  [alembic.migration] Running upgrade  -> start_neutron_vpnaas,
>>> start neutron-vpnaas chain
>>> INFO  [alembic.migration] Running upgrade start_neutron_vpnaas ->
>>> 3ea02b2a773e, add_index_tenant_id
>>> INFO  [alembic.migration] Running upgrade 3ea02b2a773e -> kilo, kilo
>>> INFO  [alembic.migration] Running upgrade kilo -> 56893333aa52, fix
>>> identifier map fk
>>>
>>
>> It seems we don't have an appropriate check for HEAD revision in at least
>> VPNaaS repo.
>> Paul and I just discussed it. We need to improve the check too.
>>
>>
>>> Should I do a separate commit that fixes the HEAD file, or just fix it
>>> as part of the bug fix I'm working on.
>>>
>>> Yes, you should immediately submit a patch to change HEAD to
>>> 56893333aa52.
>>>
>>>
>>>  BTW, at one point, after having correctly set the HEAD and versions in
>>> my new migration file, I think I ran neutron-db-manage check_migration, and
>>> I think it set the HEAD to my version, but it did that in the neutron repo,
>>> and not the VPN repo.  I might have been running from the wrong repo?
>>>
>>> I working on updating the devref docs for this process. Things have
>>> changed quite a bit with the alembic branches in separate repos.
>>>
>>>
>>>
>>>
>>>>   For my commit, I'm assuming I change the HEAD file to use my
>>>>> migration file's version?
>>>>>
>>>>
>>>>      You can do that manually too, yes.
>>>>
>>>>
>>>>>
>>>>>  I set HEAD to my migration file, and my file has a down revision of
>>>>> the previous head's revision. If I run 'neutron-db-manage --config-file
>>>>> ../neutron/etc/neutron.conf --config-file
>>>>> ../neutron/etc/neutron/plugins/ml2/ml2_conf.ini check_migration' there is
>>>>> no output so I guess that is OK.
>>>>>
>>>>>  As I develop my new migration file, is there a way that I can test
>>>>> it (running neutron-db-migration, maybe)?
>>>>>
>>>>
>>>>      When I test migrations I usually dump the database, run the
>>>> migration with neutron-db-manage upgrade HEAD (I think it's not necessary
>>>> to specify HEAD), and restore the db from the dump if the migration fails.
>>>>
>>>>
>>>>>  Is there a way to run the migration file under the debugger, as well
>>>>> (importing pdb, for example)?
>>>>>
>>>>
>>>>      The migration process is just like any python application, so I
>>>> guess you can debug it with pdb.
>>>>
>>>
>>>  @PCM Ah, so use "neutron-db-manage upgrade HEAD". That was the piece
>>> that was missing. I take it there are no specific unit tests of the
>>> migration files?
>>>
>>>
>>>
>>>>
>>>>>
>>>>>  In the migration, I can add the columns needed. What's the best way
>>>>> to fill out those fields - using raw SQL queries or create a Session object
>>>>> and access the VpnService object's router object?
>>>>>
>>>>
>>>>      If the default value for the column is not enough, and you need
>>>> to specify a value which depends on other values in the same row I would
>>>> prefer plain SQL statements, but if that become cumbersome I guess it's ok
>>>> to use sqlalchemy's session.
>>>>
>>>>
>>>>>  I see there is some op.bind() call and then engine.execute(), but
>>>>> could use some help on the best way to extract the needed queries (I need
>>>>> to access the vpnservice's router, and then access the (Port) gw_port
>>>>> relationship, and from that access the (IPAllocation) fixed_ips list).
>>>>>
>>>>
>>>>      Perhaps you can point us to the review pages on gerrit, and we
>>>> can provide detailed comments there.
>>>>
>>>
>>>  @PCM Yeah, I haven't pushed it up yet. I have a few more changes to
>>> make, and should be able to get it up in a few days. The LP bug is 1464387.
>>>
>>>  Essentially, in the vpnservices table, I'm adding an IPv4 and/or IPv6
>>> addresses for the  "local" end of VPN connections that will be established.
>>> This is to allow alternative VPN implementation (appliances, separate S/W,
>>> H/W, VM based VPN, etc) to specify addresses different than what is
>>> available on the Neutron router.
>>>
>>>  However, for the reference implementation, we'll use the Neutron
>>> router's fixed_ips list (as is done today), and to handle the migration,
>>> I'm thinking the following is needed:
>>>
>>>  1) create the new columns.
>>> 2) Identify the router for that service and obtain it's GW fixed_ips
>>> list.
>>> 3) Pick first IPv4 address (if any) and IPv6 address (if any), and store
>>> in new columns.
>>>
>>>  So I need to form a query and code to do this.
>>>
>>>
>>>
>>>>
>>>>>
>>>>>  Appreciate any advise here on how to debug the migration stuff...
>>>>>
>>>>>  Paul Michali (pc_m)
>>>>>
>>>>
>>>>      [1]
>>>> http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150707/957de913/attachment.html>