[openstack-dev] [neutron][db] online schema upgrades
Mike Bayer
mbayer at redhat.com
Thu Jun 18 13:58:50 UTC 2015
my initial proposal for scripted expand/contract migrations is up:
https://review.openstack.org/#/c/192937/
On 6/18/15 5:54 AM, Anna Kamyshnikova wrote:
>
> On Wed, Jun 17, 2015 at 8:23 PM, Mike Bayer <mbayer at redhat.com
> <mailto:mbayer at redhat.com>> wrote:
>
>
>
> On 6/17/15 12:40 PM, Ihar Hrachyshka wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> On 06/17/2015 11:27 AM, Anna Kamyshnikova wrote:
>>> Ihar, thanks for bringing this up!
>>>
>>> This is very interesting and I think it worth trying. I'm +1 on
>>> that and want to participate in this work.
>>>
>> Awesome.
>>
>>> In fact a lot *not strict* migrations are removed with
>>> juno_initial, so I hope it won't be so hard for now to apply
>>> stricter rules for migration. But what is the plan for those
>>> migrations that are *not strict* now?
>> Still, we have several live data migrations from Juno to Kilo:
>>
>> - - 14be42f3d0a5_default_sec_group_table.py: populates db with default
>> security groups;
>>
>> - - 16cdf118d31d_extra_dhcp_options_ipv6_support.py: populates
>> extraqdhcpopts with default ip_version = 4;
>>
>> - - 2d2a8a565438_hierarchical_binding.py: populates db with
>> port_binding_levels objects, then drops old tables;
>>
>> - - 35a0f3365720_add_port_security_in_ml2.py: port security field is
>> populated with True for ports and networks;
>>
>> - - 034883111f_remove_subnetpool_allow_overlap.py: drops allow_overlap
>> column from subnetpools: probably unused so we can be ok with it?..
>>
>> In any case, the plan for existing migration rules is: don't touch
>> them. Their presence in N release just indicates that we cannot get
>> online db migration in N+1. That's why we should adopt strict rules
>> the earlier the better, so that opportunity does not slip to N+X where
>> X is too far.
>>
>> The patches currently in review that look suspicious in this regard are:
>> - - I4ff7db0f5fa12b0fd1c6b10318e9177fde0210d7: moves data from one table
>> into another;
>> - - Iecb3e168a805fc5f3d59d894c3f0d9298505e872: fills new columns with
>> default server values (why is it even needed?..);
>> - - Icde55742aa78ed995bac0896c01c80c9d28aa0cf: alter_column(). Not sure
>> we are ok with it;
>> - - I66b3ee8c2f9fa6f04b9e89dc49d1a3d277d63191: probably not an issue
>> though since it touches existing live data impact rule?
>
> I made a list of migrations from juno-> kilo that are non
> expansive or do data migrations:
>
> *these contain "drop column":*
>
> 034883111f_remove_subnetpool_allow_overlap.py
> 2d2a8a565438_hierarchical_binding.py
>
> *these contain "drop table":*
>
> 28c0ffb8ebbd_remove_mlnx_plugin.py
> 2b801560a332_remove_hypervneutronplugin_tables.py
> 408cfbf6923c_remove_ryu_plugin.py
> 57086602ca0a_scrap_nsx_adv_svcs_models.py
>
> *these contain data migrations:*
>
> 14be42f3d0a5_default_sec_group_table.py
> 16cdf118d31d_extra_dhcp_options_ipv6_support.py
> 2b801560a332_remove_hypervneutronplugin_tables.py
> 2d2a8a565438_hierarchical_binding.py
> 35a0f3365720_add_port_security_in_ml2.py
>
> *Example of failure:*
>
> neutron/db/migration/alembic_migrations/versions/2d2a8a565438_hierarchical_binding.py
> - drops the following columns:
>
> op.drop_constraint(fk_name_dvr[0], 'ml2_dvr_port_bindings',
> 'foreignkey')
> op.drop_column('ml2_dvr_port_bindings', 'cap_port_filter')
> op.drop_column('ml2_dvr_port_bindings', 'segment')
> op.drop_column('ml2_dvr_port_bindings', 'driver')
>
> op.drop_constraint(fk_name[0], 'ml2_port_bindings', 'foreignkey')
> op.drop_column('ml2_port_bindings', 'driver')
> op.drop_column('ml2_port_bindings', 'segment')
> which then causes a failure in Juno:
> OperationalError: (OperationalError) (1054, "Unknown column
> 'ml2_port_bindings_1.driver' in 'field list'")
>
> In M release we will be able to create kilo_initial migration that
> will hide all these migrations(juno -> kilo), so I as there is nothing
> we can do we won't touch them as Ihar said. The problem about
> migrations that are on review is more serious, as we should adopt
> strict rules as early as possible and all core reviewer should be
> aware about that.
>
> To Ihar's list of migrations on review I can add :
> I3426b13eede8bfa29729cf3efea3419fb91175c4 - insert some data,
> I9cf36e1fd3a009c175e0d475af407a30f4e5c408 (almost ready to be merged!)
> - drop tables.
> And there are a lot changes with altering columns. One is merged in
> neutron-vpnaas.
> So we should decide these things quickly.
>
>
>
>
>
>> (the list can be incomplete)
>>> I think that we should try to use Alembic as much we could as
>>> Mike is going to support us in that and we have time to make
>>> some change in Alembic directly.
>> Yes, sure, I'm looking forward to see Mike's proposal in public.
>>> We should undoubtedly plan this work for M release because there
>>> will be some issues that will appear in the process.
>> Sure. Ihar -----BEGIN PGP SIGNATURE----- Version: GnuPG v2
>> iQEcBAEBCAAGBQJVgaL4AAoJEC5aWaUY1u57oZgH/34pgV7AqOiq4XnWOOmQ9HA9
>> jL+8E9jv8pUSW3X4v0Rm5mDuWJyWscrgy61Om+sWsmqBFAmm/gmLWm+NNADbYM5e
>> 6hsoaO5WmuvRc03MwIwsa0NEgfPc8EhT5JiZmYRjOBc85ZCs6+UOKUHBAI2EVTg8
>> t8YKdTdzxlrZQEOng1lbsUQYkHnNUZTbsREnpangfaHXBk3xmilH/ebGsz3CRUCe
>> OBrpp6q8N7mgZgK/UQKb04eS5bCna7eVmv6q7PvIO0SlYhhDbrL3+dv/SZpqQWZ/
>> Hek2Oig0IYyPygVrGc4BpT9MIaKisGoxXMn1rRB2g8us8jM58VyzqgXwEH2H4Aw=
>> =TqHb -----END PGP SIGNATURE-----
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> <mailto:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> --
> Regards,
> Ann Kamyshnikova
> Mirantis, Inc
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150618/05e44dd9/attachment.html>
More information about the OpenStack-dev
mailing list