[openstack-dev] [neutron] Plethora of dbase migration questions...
Mike Bayer
mbayer at redhat.com
Tue Jul 7 19:10:20 UTC 2015
On 7/7/15 1:28 PM, Paul Michali wrote:
> HEAD, head, 24f28869838b (my new file) all say the same thing. :(
>
>
> On Tue, Jul 7, 2015 at 12:34 PM Salvatore Orlando <sorlando at nicira.com
> <mailto:sorlando at nicira.com>> wrote:
>
> possibly I was wrong in mixing up git & alembic.
> It should be "upgrade head" - lowercase.
>
> If that doesn't work there might some other issue lurking.
>
> Salvatore
>
> On 7 July 2015 at 17:44, Paul Michali <pc at michali.net
> <mailto:pc at michali.net>> wrote:
>
> Salvatore,
>
> I changed head to the version before my new one, and then
> tried to upgrade and I see this:
> neutron-db-manage --config-file
> /opt/stack/neutron/etc/neutron.conf --service vpnaas upgrade HEAD
> Traceback (most recent call last):
> File "/usr/local/bin/neutron-db-manage", line 10, in <module>
> sys.exit(main())
> File "/opt/stack/neutron/neutron/db/migration/cli.py", line
> 238, in main
> CONF.command.func(config, CONF.command.name
> <http://CONF.command.name>)
> File "/opt/stack/neutron/neutron/db/migration/cli.py", line
> 105, in do_upgrade
> run_sanity_checks(config, revision)
> File "/opt/stack/neutron/neutron/db/migration/cli.py", line
> 229, in run_sanity_checks
> script_dir.run_env()
> File
> "/usr/local/lib/python2.7/dist-packages/alembic/script.py",
> line 390, in run_env
> util.load_python_file(self.dir, 'env.py')
> File
> "/usr/local/lib/python2.7/dist-packages/alembic/util.py", line
> 243, in load_python_file
> module = load_module_py(module_id, path)
> File
> "/usr/local/lib/python2.7/dist-packages/alembic/compat.py",
> line 79, in load_module_py
> mod = imp.load_source(module_id, path, fp)
> File
> "/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py",
> line 86, in <module>
> run_migrations_online()
> File
> "/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py",
> line 67, in run_migrations_online
> engine =
> session.create_engine(neutron_config.database.connection)
> File
> "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py",
> line 112, in create_engine
> url = sqlalchemy.engine.url.make_url(sql_connection)
> File
> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py",
> line 186, in make_url
> return _parse_rfc1738_args(name_or_url)
> File
> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py",
> line 235, in _parse_rfc1738_args
> "Could not parse rfc1738 URL from string '%s'" % name)
> sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from
> string ''
>
> Any ideas what is wrong here?
>
I'm going to guess this is the issue fixed by
https://review.openstack.org/#/c/194197/
>
> On Tue, Jul 7, 2015 at 10:05 AM Paul Michali <pc at michali.net
> <mailto:pc at michali.net>> wrote:
>
>
> Yes, I wasn't using the --service option, so I suspect
> that is why my down_version was wrong. In talking with
> Akihiro, I added a check to PEP8 and made sure that it
> fails if head is wrong. It is:
> https://review.openstack.org/#/c/199082/ (of course that
> failed py27 - I've got to see if there was some recent
> breakage in vpn repo, again).
>
> Regarding the migration, one of the new columns may be
> None, but there must be at least one IP version entry
> (there is an existing test in VPN for using a router w/o
> an external IP set). Since the new code will rely on these
> new fields, I'd like to populate them as part of the
> migration. I think it would be more complicated to handle
> during operation.
>
> Does anyone have examples of how to do queries of objects,
> from the migration upgrade() code?
>
>
> Regards,
>
> PCM
>
> On Tue, Jul 7, 2015 at 9:02 AM Akihiro Motoki
> <amotoki at gmail.com <mailto:amotoki at gmail.com>> wrote:
>
> 2015-07-07 21:39 GMT+09:00 Henry Gessau
> <gessau at cisco.com <mailto:gessau at cisco.com>>:
>
> On Tue, Jul 07, 2015, Paul Michali
> <pc at michali.net> <mailto:pc at michali.net> wrote:
>> Thanks Salvatore for the responses. See @PCM
>> in-line...
>>
>>
>>
>> On Tue, Jul 7, 2015 at 6:14 AM Salvatore Orlando
>> <sorlando at nicira.com
>> <mailto:sorlando at nicira.com>> wrote:
>>
>> Some comments inline.
>>
>> Salvatore
>>
>> On 6 July 2015 at 20:00, Paul Michali
>> <pc at michali.net <mailto:pc at michali.net>> wrote:
>>
>> Hi,
>>
>> I have some urgent requests about
>> migration that I'm hoping to get some
>> info on. I'm working on a bug where I
>> need to add two (related) fields to a
>> table for VPNaaS. Here's the objectives
>> related to migration...
>>
>> 1) create local_v4_ip and lcoal_v6_ip
>> fields in the vpnservice table
>> 2) for each entry in the vpnservice table:
>> 2.1) Get the router.gw_port.fixed_ips
>> list
>> 2.2) Determine the version of each
>> fixed IP and store the first of each
>> version (if any) into the appropriate new
>> field.
>>
>> I have created a migration file, and I
>> changed the down_revision to be the
>> number of the revision that is the first
>> in the migration chain in the VPN repo.
>>
>> Here are the many questions I have...
>>
>> When I look in the VPN repo, the HEAD
>> file has the version 'kilo', which is not
>> the current head.
>>
>>
>> Shouldn't it the version number of the
>> first file in the migration chain?
>>
>>
>> It should indeed. How are you generating the
>> revision script? Using neutron-db-manage it
>> should be updated automatically [1]
>>
>>
>> @PCM I ran neutron-db-manage, when in the neutron
>> repo, and it assigned some version, but it was
>> not the latest in the neutron-vpnaas repo.
> neutron-db-manage does not handle alembic branches
> in separate repos very well at all yet. I am
> working on updating it with
> https://review.openstack.org/198524 but I have
> quite a lot left to do.
>
>
> Yes, at now we have implicit order of running alembic
> migrations.
> First run neutron db migration and then advanced
> service migrations.
>
> I do not fully understand how alembic branch mechanism
> works, but
> I think we can have a common ancestor and have
> multiple branches,
> and each branch can evolve independently.
>
>>
>> I checked the VPN repo and there were a chain of
>> versions, which I used to determine what the head
>> should be and have set the version accordingly.
>> However, in the current repo, head is set to
>> "kilo", which appears to be incorrect. The
>> versions are:
>>
>> 56893333aa52
>> kilo <<< HEAD
>> 3ea02b2a773e
>> start_neutron_vpnaas
>> None
> Ouch. That is an error, because
> https://review.openstack.org/190569 should have
> updated HEAD but didn't.
>
> The version sequence (you can see it in any
> devstack run) is:
>
> INFO [alembic.migration] Running upgrade ->
> start_neutron_vpnaas, start neutron-vpnaas chain
> INFO [alembic.migration] Running upgrade
> start_neutron_vpnaas -> 3ea02b2a773e,
> add_index_tenant_id
> INFO [alembic.migration] Running upgrade
> 3ea02b2a773e -> kilo, kilo
> INFO [alembic.migration] Running upgrade kilo ->
> 56893333aa52, fix identifier map fk
>
>
> It seems we don't have an appropriate check for HEAD
> revision in at least VPNaaS repo.
> Paul and I just discussed it. We need to improve the
> check too.
>
>> Should I do a separate commit that fixes the HEAD
>> file, or just fix it as part of the bug fix I'm
>> working on.
> Yes, you should immediately submit a patch to
> change HEAD to 56893333aa52.
>
>>
>> BTW, at one point, after having correctly set the
>> HEAD and versions in my new migration file, I
>> think I ran neutron-db-manage check_migration,
>> and I think it set the HEAD to my version, but it
>> did that in the neutron repo, and not the VPN
>> repo. I might have been running from the wrong repo?
> I working on updating the devref docs for this
> process. Things have changed quite a bit with the
> alembic branches in separate repos.
>
>>
>>
>>
>> For my commit, I'm assuming I change the
>> HEAD file to use my migration file's version?
>>
>>
>> You can do that manually too, yes.
>>
>>
>> I set HEAD to my migration file, and my
>> file has a down revision of the previous
>> head's revision. If I run
>> 'neutron-db-manage --config-file
>> ../neutron/etc/neutron.conf --config-file
>> ../neutron/etc/neutron/plugins/ml2/ml2_conf.ini
>> check_migration' there is no output so I
>> guess that is OK.
>>
>> As I develop my new migration file, is
>> there a way that I can test it (running
>> neutron-db-migration, maybe)?
>>
>>
>> When I test migrations I usually dump the
>> database, run the migration with
>> neutron-db-manage upgrade HEAD (I think it's
>> not necessary to specify HEAD), and restore
>> the db from the dump if the migration fails.
>>
>> Is there a way to run the migration file
>> under the debugger, as well (importing
>> pdb, for example)?
>>
>>
>> The migration process is just like any python
>> application, so I guess you can debug it with
>> pdb.
>>
>>
>> @PCM Ah, so use "neutron-db-manage upgrade HEAD".
>> That was the piece that was missing. I take it
>> there are no specific unit tests of the migration
>> files?
>>
>>
>>
>> In the migration, I can add the columns
>> needed. What's the best way to fill out
>> those fields - using raw SQL queries or
>> create a Session object and access the
>> VpnService object's router object?
>>
>>
>> If the default value for the column is not
>> enough, and you need to specify a value which
>> depends on other values in the same row I
>> would prefer plain SQL statements, but if
>> that become cumbersome I guess it's ok to use
>> sqlalchemy's session.
>>
>>
>> I see there is some op.bind() call and
>> then engine.execute(), but could use some
>> help on the best way to extract the
>> needed queries (I need to access the
>> vpnservice's router, and then access the
>> (Port) gw_port relationship, and from
>> that access the (IPAllocation) fixed_ips
>> list).
>>
>>
>> Perhaps you can point us to the review pages
>> on gerrit, and we can provide detailed
>> comments there.
>>
>>
>> @PCM Yeah, I haven't pushed it up yet. I have a
>> few more changes to make, and should be able to
>> get it up in a few days. The LP bug is 1464387.
>>
>> Essentially, in the vpnservices table, I'm adding
>> an IPv4 and/or IPv6 addresses for the "local"
>> end of VPN connections that will be established.
>> This is to allow alternative VPN implementation
>> (appliances, separate S/W, H/W, VM based VPN,
>> etc) to specify addresses different than what is
>> available on the Neutron router.
>>
>> However, for the reference implementation, we'll
>> use the Neutron router's fixed_ips list (as is
>> done today), and to handle the migration, I'm
>> thinking the following is needed:
>>
>> 1) create the new columns.
>> 2) Identify the router for that service and
>> obtain it's GW fixed_ips list.
>> 3) Pick first IPv4 address (if any) and IPv6
>> address (if any), and store in new columns.
>>
>> So I need to form a query and code to do this.
>>
>>
>>
>> Appreciate any advise here on how to
>> debug the migration stuff...
>>
>> Paul Michali (pc_m)
>>
>>
>> [1]
>> http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124
>>
>>
>>
>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage
> questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage
> questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150707/605faa42/attachment-0001.html>
More information about the OpenStack-dev
mailing list