[openstack-dev] [neutron] Plethora of dbase migration questions...

Paul Michali pc at michali.net
Tue Jul 7 20:04:04 UTC 2015 

I found the issue. The upgrade is looking for config to have [database]
section and connection definition. If I use /etc/neutron/neutron.conf, then
the neutron-db-manage runs.



On Tue, Jul 7, 2015 at 3:38 PM Paul Michali <pc at michali.net> wrote:

> I have that change in the neutron repo that is being used with by this
> neutron-vpnaas repo.
>
> On Tue, Jul 7, 2015 at 3:12 PM Mike Bayer <mbayer at redhat.com> wrote:
>
>>
>>
>> On 7/7/15 1:28 PM, Paul Michali wrote:
>>
>> HEAD, head, 24f28869838b (my new file) all say the same thing. :(
>>
>>
>>  On Tue, Jul 7, 2015 at 12:34 PM Salvatore Orlando <sorlando at nicira.com>
>> wrote:
>>
>>> possibly I was wrong in mixing up git & alembic.
>>> It should be "upgrade head" - lowercase.
>>>
>>>  If that doesn't work there might some other issue lurking.
>>>
>>>  Salvatore
>>>
>>> On 7 July 2015 at 17:44, Paul Michali <pc at michali.net> wrote:
>>>
>>>> Salvatore,
>>>>
>>>>  I changed head to the version before my new one, and then tried to
>>>> upgrade and I see this:
>>>>   neutron-db-manage --config-file /opt/stack/neutron/etc/neutron.conf
>>>> --service vpnaas upgrade HEAD
>>>> Traceback (most recent call last):
>>>>   File "/usr/local/bin/neutron-db-manage", line 10, in <module>
>>>>     sys.exit(main())
>>>>   File "/opt/stack/neutron/neutron/db/migration/cli.py", line 238, in
>>>> main
>>>>     CONF.command.func(config, CONF.command.name)
>>>>   File "/opt/stack/neutron/neutron/db/migration/cli.py", line 105, in
>>>> do_upgrade
>>>>     run_sanity_checks(config, revision)
>>>>   File "/opt/stack/neutron/neutron/db/migration/cli.py", line 229, in
>>>> run_sanity_checks
>>>>     script_dir.run_env()
>>>>   File "/usr/local/lib/python2.7/dist-packages/alembic/script.py", line
>>>> 390, in run_env
>>>>     util.load_python_file(self.dir, 'env.py')
>>>>   File "/usr/local/lib/python2.7/dist-packages/alembic/util.py", line
>>>> 243, in load_python_file
>>>>     module = load_module_py(module_id, path)
>>>>   File "/usr/local/lib/python2.7/dist-packages/alembic/compat.py", line
>>>> 79, in load_module_py
>>>>     mod = imp.load_source(module_id, path, fp)
>>>>   File
>>>> "/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py",
>>>> line 86, in <module>
>>>>     run_migrations_online()
>>>>   File
>>>> "/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py",
>>>> line 67, in run_migrations_online
>>>>     engine = session.create_engine(neutron_config.database.connection)
>>>>   File
>>>> "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py",
>>>> line 112, in create_engine
>>>>     url = sqlalchemy.engine.url.make_url(sql_connection)
>>>>   File
>>>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py", line
>>>> 186, in make_url
>>>>     return _parse_rfc1738_args(name_or_url)
>>>>   File
>>>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py", line
>>>> 235, in _parse_rfc1738_args
>>>>     "Could not parse rfc1738 URL from string '%s'" % name)
>>>> sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string ''
>>>>
>>>>  Any ideas what is wrong here?
>>>>
>>>
>> I'm going to guess this is the issue fixed by
>> https://review.openstack.org/#/c/194197/
>>
>>
>>
>>
>>
>>
>>>>  On Tue, Jul 7, 2015 at 10:05 AM Paul Michali <pc at michali.net> wrote:
>>>>
>>>>>
>>>>>  Yes, I wasn't using the --service option, so I suspect that is why
>>>>> my down_version was wrong.  In talking with Akihiro, I added a check to
>>>>> PEP8 and made sure that it fails if head is wrong. It is:
>>>>> https://review.openstack.org/#/c/199082/ (of course that failed py27
>>>>> - I've got to see if there was some recent breakage in vpn repo, again).
>>>>>
>>>>>  Regarding the migration, one of the new columns may be None, but
>>>>> there must be at least one IP version entry (there is an existing test in
>>>>> VPN for using a router w/o an external IP set). Since the new code will
>>>>> rely on these new fields, I'd like to populate them as part of the
>>>>> migration. I think it would be more complicated to handle during operation.
>>>>>
>>>>>  Does anyone have examples of how to do queries of objects, from the
>>>>> migration upgrade() code?
>>>>>
>>>>>
>>>>>  Regards,
>>>>>
>>>>>  PCM
>>>>>
>>>>>  On Tue, Jul 7, 2015 at 9:02 AM Akihiro Motoki <amotoki at gmail.com>
>>>>> wrote:
>>>>>
>>>>>>  2015-07-07 21:39 GMT+09:00 Henry Gessau <gessau at cisco.com>:
>>>>>>
>>>>>>>  On Tue, Jul 07, 2015, Paul Michali <pc at michali.net>
>>>>>>> <pc at michali.net> <pc at michali.net> wrote:
>>>>>>>
>>>>>>> Thanks Salvatore for the responses. See @PCM in-line...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>  On Tue, Jul 7, 2015 at 6:14 AM Salvatore Orlando <
>>>>>>> sorlando at nicira.com> wrote:
>>>>>>>
>>>>>>>> Some comments inline.
>>>>>>>>
>>>>>>>>  Salvatore
>>>>>>>>
>>>>>>>>    On 6 July 2015 at 20:00, Paul Michali <pc at michali.net> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>>  I have some urgent requests about migration that I'm hoping to
>>>>>>>>> get some info on. I'm working on a bug where I need to add two (related)
>>>>>>>>> fields to a table for VPNaaS. Here's the objectives related to migration...
>>>>>>>>>
>>>>>>>>>  1) create local_v4_ip and lcoal_v6_ip fields in the vpnservice
>>>>>>>>> table
>>>>>>>>> 2) for each entry in the vpnservice table:
>>>>>>>>>     2.1) Get the router.gw_port.fixed_ips list
>>>>>>>>>     2.2) Determine the version of each fixed IP and store the
>>>>>>>>> first of each version (if any) into the appropriate new field.
>>>>>>>>>
>>>>>>>>>  I have created a migration file, and I changed the down_revision
>>>>>>>>> to be the number of the revision that is the first in the migration chain
>>>>>>>>> in the VPN repo.
>>>>>>>>>
>>>>>>>>>  Here are the many questions I have...
>>>>>>>>>
>>>>>>>>>  When I look in the VPN repo, the HEAD file has the version
>>>>>>>>> 'kilo', which is not the current head.
>>>>>>>>>
>>>>>>>>
>>>>>>>>>  Shouldn't it the version number of the first file in the
>>>>>>>>> migration chain?
>>>>>>>>>
>>>>>>>>
>>>>>>>>      It should indeed. How are you generating the revision script?
>>>>>>>> Using neutron-db-manage it should be updated automatically [1]
>>>>>>>>
>>>>>>>
>>>>>>>  @PCM I ran neutron-db-manage, when in the neutron repo, and it
>>>>>>> assigned some version, but it was not the latest in the neutron-vpnaas repo.
>>>>>>>
>>>>>>>  neutron-db-manage does not handle alembic branches in separate
>>>>>>> repos very well at all yet. I am working on updating it with
>>>>>>> https://review.openstack.org/198524 but I have quite a lot left to
>>>>>>> do.
>>>>>>>
>>>>>>
>>>>>>     Yes, at now we have implicit order of running alembic migrations.
>>>>>> First run neutron db migration and then advanced service migrations.
>>>>>>
>>>>>>  I do not fully understand how alembic branch mechanism works, but
>>>>>> I think we can have a common ancestor and have multiple branches,
>>>>>> and each branch can evolve independently.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>  I checked the VPN repo and there were a chain of versions, which I
>>>>>>> used to determine what the head should be and have set the version
>>>>>>> accordingly. However, in the current repo, head is set to "kilo", which
>>>>>>> appears to be incorrect.  The versions are:
>>>>>>>
>>>>>>>  56893333aa52
>>>>>>> kilo   <<< HEAD
>>>>>>> 3ea02b2a773e
>>>>>>> start_neutron_vpnaas
>>>>>>> None
>>>>>>>
>>>>>>>  Ouch. That is an error, because https://review.openstack.org/190569
>>>>>>> should have updated HEAD but didn't.
>>>>>>>
>>>>>>> The version sequence (you can see it in any devstack run) is:
>>>>>>>
>>>>>>> INFO  [alembic.migration] Running upgrade  -> start_neutron_vpnaas,
>>>>>>> start neutron-vpnaas chain
>>>>>>> INFO  [alembic.migration] Running upgrade start_neutron_vpnaas ->
>>>>>>> 3ea02b2a773e, add_index_tenant_id
>>>>>>> INFO  [alembic.migration] Running upgrade 3ea02b2a773e -> kilo, kilo
>>>>>>> INFO  [alembic.migration] Running upgrade kilo -> 56893333aa52, fix
>>>>>>> identifier map fk
>>>>>>>
>>>>>>
>>>>>>     It seems we don't have an appropriate check for HEAD revision in
>>>>>> at least VPNaaS repo.
>>>>>> Paul and I just discussed it. We need to improve the check too.
>>>>>>
>>>>>>
>>>>>>>    Should I do a separate commit that fixes the HEAD file, or just
>>>>>>> fix it as part of the bug fix I'm working on.
>>>>>>>
>>>>>>>  Yes, you should immediately submit a patch to change HEAD to
>>>>>>> 56893333aa52.
>>>>>>>
>>>>>>>
>>>>>>>  BTW, at one point, after having correctly set the HEAD and
>>>>>>> versions in my new migration file, I think I ran neutron-db-manage
>>>>>>> check_migration, and I think it set the HEAD to my version, but it did that
>>>>>>> in the neutron repo, and not the VPN repo.  I might have been running from
>>>>>>> the wrong repo?
>>>>>>>
>>>>>>>  I working on updating the devref docs for this process. Things have
>>>>>>> changed quite a bit with the alembic branches in separate repos.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>   For my commit, I'm assuming I change the HEAD file to use my
>>>>>>>>> migration file's version?
>>>>>>>>>
>>>>>>>>
>>>>>>>>      You can do that manually too, yes.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>  I set HEAD to my migration file, and my file has a down revision
>>>>>>>>> of the previous head's revision. If I run 'neutron-db-manage --config-file
>>>>>>>>> ../neutron/etc/neutron.conf --config-file
>>>>>>>>> ../neutron/etc/neutron/plugins/ml2/ml2_conf.ini check_migration' there is
>>>>>>>>> no output so I guess that is OK.
>>>>>>>>>
>>>>>>>>>  As I develop my new migration file, is there a way that I can
>>>>>>>>> test it (running neutron-db-migration, maybe)?
>>>>>>>>>
>>>>>>>>
>>>>>>>>      When I test migrations I usually dump the database, run the
>>>>>>>> migration with neutron-db-manage upgrade HEAD (I think it's not necessary
>>>>>>>> to specify HEAD), and restore the db from the dump if the migration fails.
>>>>>>>>
>>>>>>>>
>>>>>>>>>  Is there a way to run the migration file under the debugger, as
>>>>>>>>> well (importing pdb, for example)?
>>>>>>>>>
>>>>>>>>
>>>>>>>>      The migration process is just like any python application, so
>>>>>>>> I guess you can debug it with pdb.
>>>>>>>>
>>>>>>>
>>>>>>>  @PCM Ah, so use "neutron-db-manage upgrade HEAD". That was the
>>>>>>> piece that was missing. I take it there are no specific unit tests of the
>>>>>>> migration files?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>  In the migration, I can add the columns needed. What's the best
>>>>>>>>> way to fill out those fields - using raw SQL queries or create a Session
>>>>>>>>> object and access the VpnService object's router object?
>>>>>>>>>
>>>>>>>>
>>>>>>>>      If the default value for the column is not enough, and you
>>>>>>>> need to specify a value which depends on other values in the same row I
>>>>>>>> would prefer plain SQL statements, but if that become cumbersome I guess
>>>>>>>> it's ok to use sqlalchemy's session.
>>>>>>>>
>>>>>>>>
>>>>>>>>>  I see there is some op.bind() call and then engine.execute(),
>>>>>>>>> but could use some help on the best way to extract the needed queries (I
>>>>>>>>> need to access the vpnservice's router, and then access the (Port) gw_port
>>>>>>>>> relationship, and from that access the (IPAllocation) fixed_ips list).
>>>>>>>>>
>>>>>>>>
>>>>>>>>      Perhaps you can point us to the review pages on gerrit, and
>>>>>>>> we can provide detailed comments there.
>>>>>>>>
>>>>>>>
>>>>>>>  @PCM Yeah, I haven't pushed it up yet. I have a few more changes
>>>>>>> to make, and should be able to get it up in a few days. The LP bug is
>>>>>>> 1464387.
>>>>>>>
>>>>>>>  Essentially, in the vpnservices table, I'm adding an IPv4 and/or
>>>>>>> IPv6 addresses for the  "local" end of VPN connections that will be
>>>>>>> established. This is to allow alternative VPN implementation (appliances,
>>>>>>> separate S/W, H/W, VM based VPN, etc) to specify addresses different than
>>>>>>> what is available on the Neutron router.
>>>>>>>
>>>>>>>  However, for the reference implementation, we'll use the Neutron
>>>>>>> router's fixed_ips list (as is done today), and to handle the migration,
>>>>>>> I'm thinking the following is needed:
>>>>>>>
>>>>>>>  1) create the new columns.
>>>>>>> 2) Identify the router for that service and obtain it's GW fixed_ips
>>>>>>> list.
>>>>>>> 3) Pick first IPv4 address (if any) and IPv6 address (if any), and
>>>>>>> store in new columns.
>>>>>>>
>>>>>>>  So I need to form a query and code to do this.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>  Appreciate any advise here on how to debug the migration stuff...
>>>>>>>>>
>>>>>>>>>  Paul Michali (pc_m)
>>>>>>>>>
>>>>>>>>
>>>>>>>>      [1]
>>>>>>>> http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> __________________________________________________________________________
>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>> Unsubscribe:
>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>>
>>>>>> __________________________________________________________________________
>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>> Unsubscribe:
>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150707/fc7d9683/attachment.html>

More information about the OpenStack-dev mailing list