Open Stack

Hi Morgan,

Comments below..

Thanks,
Octave

On 2/6/2017 1:04 PM, Morgan Fainberg wrote:
>
>
> On Thu, Feb 2, 2017 at 2:28 PM, Octave J. Orgeron 
> <octave.orgeron at oracle.com <mailto:octave.orgeron at oracle.com>> wrote:
>
>     That refers to the total length of the row. InnoDB has a limit of
>     65k and NDB is limited to 14k.
>
>     A simple example would be the volumes table in Cinder where the
>     row length goes beyond 14k. So in the IF logic block, I change
>     columns types that are vastly oversized such as status and
>     attach_status, which by default are 255 chars. So to determine a
>     more appropriate size, I look through the Cinder code to find
>     where the possible options/states are for those columns. Then I
>     cut it down to a more reasonable size. I'm very careful when I cut
>     the size of a string column to ensure that all of the possible
>     values can be contained.
>
>     In cases where a column is extremely large for capturing the
>     outputs of a command, I will change the type to Text or TinyText
>     depending on the length required. A good example of this is in the
>     agents table of Neutron where there is a column for configurations
>     that has a string length of 4096 characters, which I change to
>     Text. Text blobs are stored differently and do not count against
>     the row length.
>
>
> So 
> https://github.com/openstack/keystone/blob/master/keystone/common/sql/core.py#L117 
> would not be an issue with the 14k limit, simply limits for things 
> such as VARCHAR would be affected (in other words, we wouldn't need to 
> change keystone's implementation since we already use sql.text here)?

Correct. Having done these patches for Kilo and Mitaka, I can say that 
Keystone has been the easiest to patch up. I haven't had to make any 
column changes at all. All I've had to do is change the mysql_engine 
setting for each table to use the value from mysql_storage_engine. So it 
hasn't had any impact on the table schema or structure.


>
>     I've also observed differences between Kilo, Mitaka, and tip where
>     even for InnoDB some of these tables are getting wider than can be
>     supported. So in the case of Cinder, some of the columns have been
>     shifted to separate tables to fit within 65k. I've seen the same
>     thing in Neutron. So I fully expect that some of the services that
>     have table bloat will have to cut the lengths or break the tables
>     up over time anyways. As that happens, it reduces the amount of
>     work for me, which is a good thing.
>
>     The most complicated database schemas to patch up are cinder,
>     glance, neutron, and nova due to the size and complexity of their
>     tables. Those also have a lot of churn between releases where the
>     schema changes more often. Other services like keystone, heat, and
>     ironic are considerably easier to work with and have well laid out
>     tables that don't change much.
>
>
> FTR: Keystone also supports "no-downtime-upgrades" (just pending some 
> functional tests before we apply for the tag) and we will be looking 
> to move towards Alembic, so make sure that the code supplied can 
> easily be swapped out between SQL-A-Migrate and Alembic (IIRC most 
> projects want to move to alembic, but it is varying levels of 
> difficulty to do so and therefore different priorities).

There are some things that I do like about Alembic and the way that it 
heals the database. But there will probably be some tricky conversions 
going from SQL Alchemy.

>
> I look forward to solid NDB support; having using NDB in the past to 
> support another project, I always thought it could be an interesting 
> choice to back OpenStack (++ to what Monty said eariler).

Thanks! I think the benefits will outweigh the investment for everyone.

>
>     Thanks,
>     Octave
>
>
>     On 2/2/2017 1:25 PM, Mike Bayer wrote:
>>
>>
>>     On 02/02/2017 02:52 PM, Mike Bayer wrote:
>>>
>>>     But more critically I noticed you referred to altering the names of
>>>     columns to suit NDB.  How will this be accomplished?   Changing
>>>     a column
>>>     name in an openstack application is no longer trivial, because
>>>     online
>>>     upgrades must be supported for applications like Nova and
>>>     Neutron.  A
>>>     column name can't just change to a new name, both columns have
>>>     to exist
>>>     and logic must be added to keep these columns synchronized.
>>>
>>
>>     correction, the phrase was "Row character length limits 65k ->
>>     14k" - does this refer to the total size of a row?  I guess rows
>>     that store JSON or tables like keystone tokens are what you had
>>     in mind here, can you give specifics ?
>>
>>
>>
>>     __________________________________________________________________________
>>
>>     OpenStack Development Mailing List (not for usage questions)
>>     Unsubscribe:
>>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>     <mailto:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe>
>>
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>
>     -- 
>
>     Oracle <http://www.oracle.com/>
>     Octave J. Orgeron | Sr. Principal Architect and Software Engineer
>     Oracle Linux OpenStack
>     500 Eldorado Blvd. | Broomfield, CO 80021
>     Certified Oracle Enterprise Architect: Systems Infrastructure
>     <http://www.oracle.com/us/solutions/enterprise-architecture/index.html>
>     Green Oracle <http://www.oracle.com/commitment> Oracle is
>     committed to developing practices and products that help protect
>     the environment
>
>
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 

Oracle <http://www.oracle.com/>
Octave J. Orgeron | Sr. Principal Architect and Software Engineer
Oracle Linux OpenStack
<tel:+17206161550> 500 Eldorado Blvd. | Broomfield, CO 80021
Certified Oracle Enterprise Architect: Systems Infrastructure 
<http://www.oracle.com/us/solutions/enterprise-architecture/index.html>
Green Oracle <http://www.oracle.com/commitment> Oracle is committed to 
developing practices and products that help protect the environment

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170206/0ae9a27a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 658 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170206/0ae9a27a/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 2398 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170206/0ae9a27a/attachment.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 356 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170206/0ae9a27a/attachment-0001.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: oracle_sig_logo.gif
Type: image/gif
Size: 658 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170206/0ae9a27a/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: oea_logo.jpg
Type: image/jpeg
Size: 2398 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170206/0ae9a27a/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: green-for-email-sig_0.gif
Type: image/gif
Size: 356 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170206/0ae9a27a/attachment-0003.gif>