[openstack-dev] [neutron][db] online schema upgrades

Mike Bayer mbayer at redhat.com
Thu Jun 18 13:58:50 UTC 2015


my initial proposal for scripted expand/contract migrations is up:

https://review.openstack.org/#/c/192937/




On 6/18/15 5:54 AM, Anna Kamyshnikova wrote:
>
> On Wed, Jun 17, 2015 at 8:23 PM, Mike Bayer <mbayer at redhat.com 
> <mailto:mbayer at redhat.com>> wrote:
>
>
>
>     On 6/17/15 12:40 PM, Ihar Hrachyshka wrote:
>>     -----BEGIN PGP SIGNED MESSAGE-----
>>     Hash: SHA256
>>
>>     On 06/17/2015 11:27 AM, Anna Kamyshnikova wrote:
>>>     Ihar, thanks for bringing this up!
>>>
>>>     This is very interesting and I think it worth trying. I'm +1 on
>>>     that and want to participate in this work.
>>>
>>     Awesome.
>>
>>>     In fact a lot *not strict* migrations are removed with
>>>     juno_initial, so I hope it won't be so hard for now to apply
>>>     stricter rules for migration. But what is the plan for those
>>>     migrations that are *not strict* now?
>>     Still, we have several live data migrations from Juno to Kilo:
>>
>>     - - 14be42f3d0a5_default_sec_group_table.py: populates db with default
>>     security groups;
>>
>>     - - 16cdf118d31d_extra_dhcp_options_ipv6_support.py: populates
>>     extraqdhcpopts with default ip_version = 4;
>>
>>     - - 2d2a8a565438_hierarchical_binding.py: populates db with
>>     port_binding_levels objects, then drops old tables;
>>
>>     - - 35a0f3365720_add_port_security_in_ml2.py: port security field is
>>     populated with True for ports and networks;
>>
>>     - - 034883111f_remove_subnetpool_allow_overlap.py: drops allow_overlap
>>     column from subnetpools: probably unused so we can be ok with it?..
>>
>>     In any case, the plan for existing migration rules is: don't touch
>>     them. Their presence in N release just indicates that we cannot get
>>     online db migration in N+1. That's why we should adopt strict rules
>>     the earlier the better, so that opportunity does not slip to N+X where
>>     X is too far.
>>
>>     The patches currently in review that look suspicious in this regard are:
>>     - - I4ff7db0f5fa12b0fd1c6b10318e9177fde0210d7: moves data from one table
>>     into another;
>>     - - Iecb3e168a805fc5f3d59d894c3f0d9298505e872: fills new columns with
>>     default server values (why is it even needed?..);
>>     - - Icde55742aa78ed995bac0896c01c80c9d28aa0cf: alter_column(). Not sure
>>     we are ok with it;
>>     - - I66b3ee8c2f9fa6f04b9e89dc49d1a3d277d63191: probably not an issue
>>     though since it touches existing live data impact rule?
>
>     I made a list of migrations from juno-> kilo that are non
>     expansive or do data migrations:
>
>     *these contain "drop column":*
>
>     034883111f_remove_subnetpool_allow_overlap.py
>     2d2a8a565438_hierarchical_binding.py
>
>     *these contain "drop table":*
>
>     28c0ffb8ebbd_remove_mlnx_plugin.py
>     2b801560a332_remove_hypervneutronplugin_tables.py
>     408cfbf6923c_remove_ryu_plugin.py
>     57086602ca0a_scrap_nsx_adv_svcs_models.py
>
>     *these contain data migrations:*
>
>     14be42f3d0a5_default_sec_group_table.py
>     16cdf118d31d_extra_dhcp_options_ipv6_support.py
>     2b801560a332_remove_hypervneutronplugin_tables.py
>     2d2a8a565438_hierarchical_binding.py
>     35a0f3365720_add_port_security_in_ml2.py
>
>     *Example of failure:*
>
>     neutron/db/migration/alembic_migrations/versions/2d2a8a565438_hierarchical_binding.py
>     - drops the following columns:
>
>     op.drop_constraint(fk_name_dvr[0], 'ml2_dvr_port_bindings',
>     'foreignkey')
>     op.drop_column('ml2_dvr_port_bindings', 'cap_port_filter')
>     op.drop_column('ml2_dvr_port_bindings', 'segment')
>     op.drop_column('ml2_dvr_port_bindings', 'driver')
>
>     op.drop_constraint(fk_name[0], 'ml2_port_bindings', 'foreignkey')
>     op.drop_column('ml2_port_bindings', 'driver')
>     op.drop_column('ml2_port_bindings', 'segment')
>         which then causes a failure in Juno:
>         OperationalError: (OperationalError) (1054, "Unknown column
>     'ml2_port_bindings_1.driver' in 'field list'")
>
> In M release we will be able to create kilo_initial migration that 
> will hide all these migrations(juno -> kilo), so I as there is nothing 
> we can do we won't touch them as Ihar said. The problem about 
> migrations that are on review is more serious, as we should adopt 
> strict rules as early as possible and all core reviewer should be 
> aware about that.
>
> To Ihar's list of migrations on review  I can add :
> I3426b13eede8bfa29729cf3efea3419fb91175c4 - insert some data,
> I9cf36e1fd3a009c175e0d475af407a30f4e5c408 (almost ready to be merged!) 
> - drop tables.
> And there are a lot changes with altering columns. One is merged in 
> neutron-vpnaas.
> So we should decide these things quickly.
>
>
>
>
>
>>     (the list can be incomplete)
>>>     I think that we should try to use Alembic as much we could as
>>>     Mike is going to support us in that and we have time to make
>>>     some change in Alembic directly.
>>     Yes, sure, I'm looking forward to see Mike's proposal in public.
>>>     We should undoubtedly plan this work for M release because there
>>>     will be some issues that will appear in the process.
>>     Sure. Ihar -----BEGIN PGP SIGNATURE----- Version: GnuPG v2
>>     iQEcBAEBCAAGBQJVgaL4AAoJEC5aWaUY1u57oZgH/34pgV7AqOiq4XnWOOmQ9HA9
>>     jL+8E9jv8pUSW3X4v0Rm5mDuWJyWscrgy61Om+sWsmqBFAmm/gmLWm+NNADbYM5e
>>     6hsoaO5WmuvRc03MwIwsa0NEgfPc8EhT5JiZmYRjOBc85ZCs6+UOKUHBAI2EVTg8
>>     t8YKdTdzxlrZQEOng1lbsUQYkHnNUZTbsREnpangfaHXBk3xmilH/ebGsz3CRUCe
>>     OBrpp6q8N7mgZgK/UQKb04eS5bCna7eVmv6q7PvIO0SlYhhDbrL3+dv/SZpqQWZ/
>>     Hek2Oig0IYyPygVrGc4BpT9MIaKisGoxXMn1rRB2g8us8jM58VyzqgXwEH2H4Aw=
>>     =TqHb -----END PGP SIGNATURE-----
>>     __________________________________________________________________________
>>     OpenStack Development Mailing List (not for usage questions)
>>     Unsubscribe:
>>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>     <mailto:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe>
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> -- 
> Regards,
> Ann Kamyshnikova
> Mirantis, Inc
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150618/05e44dd9/attachment.html>


More information about the OpenStack-dev mailing list