Open Stack

Comments below..

On 2/3/2017 8:34 AM, Mike Bayer wrote:
>
>
> On 02/03/2017 10:21 AM, Doug Hellmann wrote:
>> Excerpts from Mike Bayer's message of 2017-02-03 09:41:11 -0500:
>>>
>>> On 02/02/2017 05:28 PM, Octave J. Orgeron wrote:
>>>> That refers to the total length of the row. InnoDB has a limit of 65k
>>>> and NDB is limited to 14k.
>>>>
>>>> A simple example would be the volumes table in Cinder where the row
>>>> length goes beyond 14k. So in the IF logic block, I change columns 
>>>> types
>>>> that are vastly oversized such as status and attach_status, which by
>>>> default are 255 chars.
>>>
>>>
>>> let me give you a tip on IF blocks, that they are a bit of an
>>> anti-pattern.  If you want a column type to do one thing in one case,
>>> and another in another case, create an object that does the thing 
>>> you want:
>>>
>>>
>>> some_table = Table(
>>>      'some_table', metadata,
>>>      Column('my_column', VARCHAR(255).with_variant(VARCHAR(50), 'ndb'))
>>> )
>>
>> I wonder if we want to do either, though. Shouldn't we try to use
>> the same (smaller) column size all the time? Otherwise we end up
>> with another incompatibility between different deployments, since
>> sometimes things like names might have different sizes in different
>> clouds.
>
> in that case you have to do a migration which as you know these days 
> means the "old" column remains for a whole release cycle and the 
> application must undergo significant complexity, either at the app 
> level or in triggers, to keep data between "old" and "new" columns 
> simultaneously.   So one advantage to keeping this at the "create for 
> NDB" level is that we don't need to get into schema migrations.
>
> Unless we changed the value in the application and its migration files 
> completely, and *didnt* migrate old applications, and just hope/ensure 
> that they aren't writing larger data values.   Maybe that's possible 
> though it seems a little scary.   Perhaps some kind of annotated type 
> like VARCHAR(50, unmigrated=255) to note what's going on.
>
>

This is one of the things that I worried about and why I took the 
approach of doing the logic for NDB and keeping the default logic there 
based on the mysql_storage_engine setting. Perhaps it makes sense to do 
things this way first and then in a future release do the migrations of 
the column sizes and types as a second phase. Then as a third phase we 
can remove the column size and type logic changes? At that point, the 
only real change left will be the substitution of the 
"mysql_engine=InnoDB" with a value that we could abstract from somewhere 
(oslo.db or a dialect stub).

As for the foreign key, constraints, and index ordering, it's good 
practice to do things in the right order and doesn't impact any 
functionality in InnoDB. So that's actually a plus that I'll fix those 
with my patches.

Then that will just leave the logic for dealing with savepoints and 
nested operations. Then when NDB adds those features, we can drop that 
logic down the road.

>
>
>>
>>> I think we might want to look into creating a stub dialect called 'ndb'
>>> that subclasses mysql+pymysql.   Treating ndb as a whole different
>>> database means there's no longer the need for a flag in oslo.db, the
>>> 'ndb' name would instead be interpreted as a new backend - the main
>>> thing would be ensuring all the mysql-appropriate hooks in oslo.db are
>>> also emitted for ndb, but this also gives us a way to pick and choose
>>> which hooks apply.   It seems like there may be enough different about
>>> it to separate it at this level.
>>>
>>> Not sure if people on the list are seeing that we are simultaneously
>>> talking about getting rid of Postgresql in the efforts to support only
>>> "one database", while at the same time adding one that is in many 
>>> ways a
>>> new database.
>>
>> Yes, that does seem a bit ironic. That's also why I was pointing
>> out that we're going to want to have people lined up to support the
>> work before starting. The lack of help with Postresql testing
>> resulted in removing it from the gate, and possibly to dropping
>> support entirely.
>>
>> For reference, the discussion in [1] led to this proposed TC
>> resolution [2].
>>
>> [1] 
>> http://lists.openstack.org/pipermail/openstack-dev/2017-February/thread.html#111357
>> [2] https://review.openstack.org/427880
>>
>>>
>>>
>>>
>>>
>>> So to determine a more appropriate size, I look
>>>> through the Cinder code to find where the possible options/states are
>>>> for those columns. Then I cut it down to a more reasonable size. I'm
>>>> very careful when I cut the size of a string column to ensure that all
>>>> of the possible values can be contained.
>>>>
>>>> In cases where a column is extremely large for capturing the 
>>>> outputs of
>>>> a command, I will change the type to Text or TinyText depending on the
>>>> length required. A good example of this is in the agents table of
>>>> Neutron where there is a column for configurations that has a string
>>>> length of 4096 characters, which I change to Text. Text blobs are 
>>>> stored
>>>> differently and do not count against the row length.
>>>>
>>>> I've also observed differences between Kilo, Mitaka, and tip where 
>>>> even
>>>> for InnoDB some of these tables are getting wider than can be 
>>>> supported.
>>>> So in the case of Cinder, some of the columns have been shifted to
>>>> separate tables to fit within 65k. I've seen the same thing in 
>>>> Neutron.
>>>> So I fully expect that some of the services that have table bloat will
>>>> have to cut the lengths or break the tables up over time anyways. As
>>>> that happens, it reduces the amount of work for me, which is a good 
>>>> thing.
>>>>
>>>> The most complicated database schemas to patch up are cinder, glance,
>>>> neutron, and nova due to the size and complexity of their tables. 
>>>> Those
>>>> also have a lot of churn between releases where the schema changes 
>>>> more
>>>> often. Other services like keystone, heat, and ironic are considerably
>>>> easier to work with and have well laid out tables that don't change 
>>>> much.
>>>>
>>>> Thanks,
>>>> Octave
>>>>
>>>> On 2/2/2017 1:25 PM, Mike Bayer wrote:
>>>>>
>>>>>
>>>>> On 02/02/2017 02:52 PM, Mike Bayer wrote:
>>>>>>
>>>>>> But more critically I noticed you referred to altering the names of
>>>>>> columns to suit NDB.  How will this be accomplished? Changing a 
>>>>>> column
>>>>>> name in an openstack application is no longer trivial, because 
>>>>>> online
>>>>>> upgrades must be supported for applications like Nova and 
>>>>>> Neutron.  A
>>>>>> column name can't just change to a new name, both columns have to 
>>>>>> exist
>>>>>> and logic must be added to keep these columns synchronized.
>>>>>>
>>>>>
>>>>> correction, the phrase was "Row character length limits 65k -> 14k" -
>>>>> does this refer to the total size of a row?  I guess rows that store
>>>>> JSON or tables like keystone tokens are what you had in mind here, 
>>>>> can
>>>>> you give specifics ?
>>>>>
>>>>>
>>>>>
>>>>> __________________________________________________________________________ 
>>>>>
>>>>>
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>> -- 
>>>>
>>>> Oracle <http://www.oracle.com/>
>>>> Octave J. Orgeron | Sr. Principal Architect and Software Engineer
>>>> Oracle Linux OpenStack
>>>> Mobile: +1-720-616-1550 <tel:+17206161550>
>>>> 500 Eldorado Blvd. | Broomfield, CO 80021
>>>> Certified Oracle Enterprise Architect: Systems Infrastructure
>>>> <http://www.oracle.com/us/solutions/enterprise-architecture/index.html> 
>>>>
>>>> Green Oracle <http://www.oracle.com/commitment> Oracle is committed to
>>>> developing practices and products that help protect the environment
>>>>
>>>>
>>>>
>>>> __________________________________________________________________________ 
>>>>
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe: 
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>
>> __________________________________________________________________________ 
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: 
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __________________________________________________________________________ 
>
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

-- 

Oracle <http://www.oracle.com/>
Octave J. Orgeron | Sr. Principal Architect and Software Engineer
Oracle Linux OpenStack
Mobile: +1-720-616-1550 <tel:+17206161550>
500 Eldorado Blvd. | Broomfield, CO 80021
Certified Oracle Enterprise Architect: Systems Infrastructure 
<http://www.oracle.com/us/solutions/enterprise-architecture/index.html>
Green Oracle <http://www.oracle.com/commitment> Oracle is committed to 
developing practices and products that help protect the environment

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170203/26276dde/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: oracle_sig_logo.gif
Type: image/gif
Size: 658 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170203/26276dde/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: oea_logo.jpg
Type: image/jpeg
Size: 2398 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170203/26276dde/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: green-for-email-sig_0.gif
Type: image/gif
Size: 356 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170203/26276dde/attachment-0001.gif>