Open Stack

Well, I can run the upgrade command, but it doesn't seem to be processing
my new migration file. I have tried using upgrade with 'head', and with the
HEAD file set to the previous version and to the new version. In both
cases, I get these info messages twice: "Context impl MySWLImpl." and "Will
assume non-transactional DDL." I put a "import pdb; pdb.set_trace() in my
migration file, but it never reaches that.

What am I possibly missing?

Regards,

PCM


On Tue, Jul 7, 2015 at 4:04 PM Paul Michali <pc at michali.net> wrote:

> I found the issue. The upgrade is looking for config to have [database]
> section and connection definition. If I use /etc/neutron/neutron.conf, then
> the neutron-db-manage runs.
>
>
>
> On Tue, Jul 7, 2015 at 3:38 PM Paul Michali <pc at michali.net> wrote:
>
>> I have that change in the neutron repo that is being used with by this
>> neutron-vpnaas repo.
>>
>> On Tue, Jul 7, 2015 at 3:12 PM Mike Bayer <mbayer at redhat.com> wrote:
>>
>>>
>>>
>>> On 7/7/15 1:28 PM, Paul Michali wrote:
>>>
>>> HEAD, head, 24f28869838b (my new file) all say the same thing. :(
>>>
>>>
>>>  On Tue, Jul 7, 2015 at 12:34 PM Salvatore Orlando <sorlando at nicira.com>
>>> wrote:
>>>
>>>> possibly I was wrong in mixing up git & alembic.
>>>> It should be "upgrade head" - lowercase.
>>>>
>>>>  If that doesn't work there might some other issue lurking.
>>>>
>>>>  Salvatore
>>>>
>>>> On 7 July 2015 at 17:44, Paul Michali <pc at michali.net> wrote:
>>>>
>>>>> Salvatore,
>>>>>
>>>>>  I changed head to the version before my new one, and then tried to
>>>>> upgrade and I see this:
>>>>>   neutron-db-manage --config-file /opt/stack/neutron/etc/neutron.conf
>>>>> --service vpnaas upgrade HEAD
>>>>> Traceback (most recent call last):
>>>>>   File "/usr/local/bin/neutron-db-manage", line 10, in <module>
>>>>>     sys.exit(main())
>>>>>   File "/opt/stack/neutron/neutron/db/migration/cli.py", line 238, in
>>>>> main
>>>>>     CONF.command.func(config, CONF.command.name)
>>>>>   File "/opt/stack/neutron/neutron/db/migration/cli.py", line 105, in
>>>>> do_upgrade
>>>>>     run_sanity_checks(config, revision)
>>>>>   File "/opt/stack/neutron/neutron/db/migration/cli.py", line 229, in
>>>>> run_sanity_checks
>>>>>     script_dir.run_env()
>>>>>   File "/usr/local/lib/python2.7/dist-packages/alembic/script.py",
>>>>> line 390, in run_env
>>>>>     util.load_python_file(self.dir, 'env.py')
>>>>>   File "/usr/local/lib/python2.7/dist-packages/alembic/util.py", line
>>>>> 243, in load_python_file
>>>>>     module = load_module_py(module_id, path)
>>>>>   File "/usr/local/lib/python2.7/dist-packages/alembic/compat.py",
>>>>> line 79, in load_module_py
>>>>>     mod = imp.load_source(module_id, path, fp)
>>>>>   File
>>>>> "/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py",
>>>>> line 86, in <module>
>>>>>     run_migrations_online()
>>>>>   File
>>>>> "/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py",
>>>>> line 67, in run_migrations_online
>>>>>     engine = session.create_engine(neutron_config.database.connection)
>>>>>   File
>>>>> "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py",
>>>>> line 112, in create_engine
>>>>>     url = sqlalchemy.engine.url.make_url(sql_connection)
>>>>>   File
>>>>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py", line
>>>>> 186, in make_url
>>>>>     return _parse_rfc1738_args(name_or_url)
>>>>>   File
>>>>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py", line
>>>>> 235, in _parse_rfc1738_args
>>>>>     "Could not parse rfc1738 URL from string '%s'" % name)
>>>>> sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string
>>>>> ''
>>>>>
>>>>>  Any ideas what is wrong here?
>>>>>
>>>>
>>> I'm going to guess this is the issue fixed by
>>> https://review.openstack.org/#/c/194197/
>>>
>>>
>>>
>>>
>>>
>>>
>>>>>  On Tue, Jul 7, 2015 at 10:05 AM Paul Michali <pc at michali.net> wrote:
>>>>>
>>>>>>
>>>>>>  Yes, I wasn't using the --service option, so I suspect that is why
>>>>>> my down_version was wrong.  In talking with Akihiro, I added a check to
>>>>>> PEP8 and made sure that it fails if head is wrong. It is:
>>>>>> https://review.openstack.org/#/c/199082/ (of course that failed py27
>>>>>> - I've got to see if there was some recent breakage in vpn repo, again).
>>>>>>
>>>>>>  Regarding the migration, one of the new columns may be None, but
>>>>>> there must be at least one IP version entry (there is an existing test in
>>>>>> VPN for using a router w/o an external IP set). Since the new code will
>>>>>> rely on these new fields, I'd like to populate them as part of the
>>>>>> migration. I think it would be more complicated to handle during operation.
>>>>>>
>>>>>>  Does anyone have examples of how to do queries of objects, from the
>>>>>> migration upgrade() code?
>>>>>>
>>>>>>
>>>>>>  Regards,
>>>>>>
>>>>>>  PCM
>>>>>>
>>>>>>  On Tue, Jul 7, 2015 at 9:02 AM Akihiro Motoki <amotoki at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>  2015-07-07 21:39 GMT+09:00 Henry Gessau <gessau at cisco.com>:
>>>>>>>
>>>>>>>>  On Tue, Jul 07, 2015, Paul Michali <pc at michali.net>
>>>>>>>> <pc at michali.net> <pc at michali.net> wrote:
>>>>>>>>
>>>>>>>> Thanks Salvatore for the responses. See @PCM in-line...
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>  On Tue, Jul 7, 2015 at 6:14 AM Salvatore Orlando <
>>>>>>>> sorlando at nicira.com> wrote:
>>>>>>>>
>>>>>>>>> Some comments inline.
>>>>>>>>>
>>>>>>>>>  Salvatore
>>>>>>>>>
>>>>>>>>>    On 6 July 2015 at 20:00, Paul Michali <pc at michali.net> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>>  I have some urgent requests about migration that I'm hoping to
>>>>>>>>>> get some info on. I'm working on a bug where I need to add two (related)
>>>>>>>>>> fields to a table for VPNaaS. Here's the objectives related to migration...
>>>>>>>>>>
>>>>>>>>>>  1) create local_v4_ip and lcoal_v6_ip fields in the vpnservice
>>>>>>>>>> table
>>>>>>>>>> 2) for each entry in the vpnservice table:
>>>>>>>>>>     2.1) Get the router.gw_port.fixed_ips list
>>>>>>>>>>     2.2) Determine the version of each fixed IP and store the
>>>>>>>>>> first of each version (if any) into the appropriate new field.
>>>>>>>>>>
>>>>>>>>>>  I have created a migration file, and I changed the
>>>>>>>>>> down_revision to be the number of the revision that is the first in the
>>>>>>>>>> migration chain in the VPN repo.
>>>>>>>>>>
>>>>>>>>>>  Here are the many questions I have...
>>>>>>>>>>
>>>>>>>>>>  When I look in the VPN repo, the HEAD file has the version
>>>>>>>>>> 'kilo', which is not the current head.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>  Shouldn't it the version number of the first file in the
>>>>>>>>>> migration chain?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>      It should indeed. How are you generating the revision
>>>>>>>>> script? Using neutron-db-manage it should be updated automatically [1]
>>>>>>>>>
>>>>>>>>
>>>>>>>>  @PCM I ran neutron-db-manage, when in the neutron repo, and it
>>>>>>>> assigned some version, but it was not the latest in the neutron-vpnaas repo.
>>>>>>>>
>>>>>>>>  neutron-db-manage does not handle alembic branches in separate
>>>>>>>> repos very well at all yet. I am working on updating it with
>>>>>>>> https://review.openstack.org/198524 but I have quite a lot left to
>>>>>>>> do.
>>>>>>>>
>>>>>>>
>>>>>>>     Yes, at now we have implicit order of running alembic
>>>>>>> migrations.
>>>>>>> First run neutron db migration and then advanced service migrations.
>>>>>>>
>>>>>>>  I do not fully understand how alembic branch mechanism works, but
>>>>>>> I think we can have a common ancestor and have multiple branches,
>>>>>>> and each branch can evolve independently.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>  I checked the VPN repo and there were a chain of versions, which
>>>>>>>> I used to determine what the head should be and have set the version
>>>>>>>> accordingly. However, in the current repo, head is set to "kilo", which
>>>>>>>> appears to be incorrect.  The versions are:
>>>>>>>>
>>>>>>>>  56893333aa52
>>>>>>>> kilo   <<< HEAD
>>>>>>>> 3ea02b2a773e
>>>>>>>> start_neutron_vpnaas
>>>>>>>> None
>>>>>>>>
>>>>>>>>  Ouch. That is an error, because
>>>>>>>> https://review.openstack.org/190569 should have updated HEAD but
>>>>>>>> didn't.
>>>>>>>>
>>>>>>>> The version sequence (you can see it in any devstack run) is:
>>>>>>>>
>>>>>>>> INFO  [alembic.migration] Running upgrade  -> start_neutron_vpnaas,
>>>>>>>> start neutron-vpnaas chain
>>>>>>>> INFO  [alembic.migration] Running upgrade start_neutron_vpnaas ->
>>>>>>>> 3ea02b2a773e, add_index_tenant_id
>>>>>>>> INFO  [alembic.migration] Running upgrade 3ea02b2a773e -> kilo, kilo
>>>>>>>> INFO  [alembic.migration] Running upgrade kilo -> 56893333aa52, fix
>>>>>>>> identifier map fk
>>>>>>>>
>>>>>>>
>>>>>>>     It seems we don't have an appropriate check for HEAD revision
>>>>>>> in at least VPNaaS repo.
>>>>>>> Paul and I just discussed it. We need to improve the check too.
>>>>>>>
>>>>>>>
>>>>>>>>    Should I do a separate commit that fixes the HEAD file, or just
>>>>>>>> fix it as part of the bug fix I'm working on.
>>>>>>>>
>>>>>>>>  Yes, you should immediately submit a patch to change HEAD to
>>>>>>>> 56893333aa52.
>>>>>>>>
>>>>>>>>
>>>>>>>>  BTW, at one point, after having correctly set the HEAD and
>>>>>>>> versions in my new migration file, I think I ran neutron-db-manage
>>>>>>>> check_migration, and I think it set the HEAD to my version, but it did that
>>>>>>>> in the neutron repo, and not the VPN repo.  I might have been running from
>>>>>>>> the wrong repo?
>>>>>>>>
>>>>>>>>  I working on updating the devref docs for this process. Things
>>>>>>>> have changed quite a bit with the alembic branches in separate repos.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>   For my commit, I'm assuming I change the HEAD file to use my
>>>>>>>>>> migration file's version?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>      You can do that manually too, yes.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  I set HEAD to my migration file, and my file has a down
>>>>>>>>>> revision of the previous head's revision. If I run 'neutron-db-manage
>>>>>>>>>> --config-file ../neutron/etc/neutron.conf --config-file
>>>>>>>>>> ../neutron/etc/neutron/plugins/ml2/ml2_conf.ini check_migration' there is
>>>>>>>>>> no output so I guess that is OK.
>>>>>>>>>>
>>>>>>>>>>  As I develop my new migration file, is there a way that I can
>>>>>>>>>> test it (running neutron-db-migration, maybe)?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>      When I test migrations I usually dump the database, run the
>>>>>>>>> migration with neutron-db-manage upgrade HEAD (I think it's not necessary
>>>>>>>>> to specify HEAD), and restore the db from the dump if the migration fails.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>  Is there a way to run the migration file under the debugger, as
>>>>>>>>>> well (importing pdb, for example)?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>      The migration process is just like any python application,
>>>>>>>>> so I guess you can debug it with pdb.
>>>>>>>>>
>>>>>>>>
>>>>>>>>  @PCM Ah, so use "neutron-db-manage upgrade HEAD". That was the
>>>>>>>> piece that was missing. I take it there are no specific unit tests of the
>>>>>>>> migration files?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  In the migration, I can add the columns needed. What's the best
>>>>>>>>>> way to fill out those fields - using raw SQL queries or create a Session
>>>>>>>>>> object and access the VpnService object's router object?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>      If the default value for the column is not enough, and you
>>>>>>>>> need to specify a value which depends on other values in the same row I
>>>>>>>>> would prefer plain SQL statements, but if that become cumbersome I guess
>>>>>>>>> it's ok to use sqlalchemy's session.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>  I see there is some op.bind() call and then engine.execute(),
>>>>>>>>>> but could use some help on the best way to extract the needed queries (I
>>>>>>>>>> need to access the vpnservice's router, and then access the (Port) gw_port
>>>>>>>>>> relationship, and from that access the (IPAllocation) fixed_ips list).
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>      Perhaps you can point us to the review pages on gerrit, and
>>>>>>>>> we can provide detailed comments there.
>>>>>>>>>
>>>>>>>>
>>>>>>>>  @PCM Yeah, I haven't pushed it up yet. I have a few more changes
>>>>>>>> to make, and should be able to get it up in a few days. The LP bug is
>>>>>>>> 1464387.
>>>>>>>>
>>>>>>>>  Essentially, in the vpnservices table, I'm adding an IPv4 and/or
>>>>>>>> IPv6 addresses for the  "local" end of VPN connections that will be
>>>>>>>> established. This is to allow alternative VPN implementation (appliances,
>>>>>>>> separate S/W, H/W, VM based VPN, etc) to specify addresses different than
>>>>>>>> what is available on the Neutron router.
>>>>>>>>
>>>>>>>>  However, for the reference implementation, we'll use the Neutron
>>>>>>>> router's fixed_ips list (as is done today), and to handle the migration,
>>>>>>>> I'm thinking the following is needed:
>>>>>>>>
>>>>>>>>  1) create the new columns.
>>>>>>>> 2) Identify the router for that service and obtain it's GW
>>>>>>>> fixed_ips list.
>>>>>>>> 3) Pick first IPv4 address (if any) and IPv6 address (if any), and
>>>>>>>> store in new columns.
>>>>>>>>
>>>>>>>>  So I need to form a query and code to do this.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  Appreciate any advise here on how to debug the migration
>>>>>>>>>> stuff...
>>>>>>>>>>
>>>>>>>>>>  Paul Michali (pc_m)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>      [1]
>>>>>>>>> http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> __________________________________________________________________________
>>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>>> Unsubscribe:
>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>
>>>>>>>>
>>>>>>> __________________________________________________________________________
>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>> Unsubscribe:
>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>
>>>>>
>>>>> __________________________________________________________________________
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150707/8f321115/attachment.html>