[openstack-dev] [oslo][oslo.db] MySQL Cluster support
Octave J. Orgeron
octave.orgeron at oracle.com
Mon Feb 6 21:50:11 UTC 2017
Hi Morgan,
Comments below..
Thanks,
Octave
On 2/6/2017 1:04 PM, Morgan Fainberg wrote:
>
>
> On Thu, Feb 2, 2017 at 2:28 PM, Octave J. Orgeron
> <octave.orgeron at oracle.com <mailto:octave.orgeron at oracle.com>> wrote:
>
> That refers to the total length of the row. InnoDB has a limit of
> 65k and NDB is limited to 14k.
>
> A simple example would be the volumes table in Cinder where the
> row length goes beyond 14k. So in the IF logic block, I change
> columns types that are vastly oversized such as status and
> attach_status, which by default are 255 chars. So to determine a
> more appropriate size, I look through the Cinder code to find
> where the possible options/states are for those columns. Then I
> cut it down to a more reasonable size. I'm very careful when I cut
> the size of a string column to ensure that all of the possible
> values can be contained.
>
> In cases where a column is extremely large for capturing the
> outputs of a command, I will change the type to Text or TinyText
> depending on the length required. A good example of this is in the
> agents table of Neutron where there is a column for configurations
> that has a string length of 4096 characters, which I change to
> Text. Text blobs are stored differently and do not count against
> the row length.
>
>
> So
> https://github.com/openstack/keystone/blob/master/keystone/common/sql/core.py#L117
> would not be an issue with the 14k limit, simply limits for things
> such as VARCHAR would be affected (in other words, we wouldn't need to
> change keystone's implementation since we already use sql.text here)?
Correct. Having done these patches for Kilo and Mitaka, I can say that
Keystone has been the easiest to patch up. I haven't had to make any
column changes at all. All I've had to do is change the mysql_engine
setting for each table to use the value from mysql_storage_engine. So it
hasn't had any impact on the table schema or structure.
>
> I've also observed differences between Kilo, Mitaka, and tip where
> even for InnoDB some of these tables are getting wider than can be
> supported. So in the case of Cinder, some of the columns have been
> shifted to separate tables to fit within 65k. I've seen the same
> thing in Neutron. So I fully expect that some of the services that
> have table bloat will have to cut the lengths or break the tables
> up over time anyways. As that happens, it reduces the amount of
> work for me, which is a good thing.
>
> The most complicated database schemas to patch up are cinder,
> glance, neutron, and nova due to the size and complexity of their
> tables. Those also have a lot of churn between releases where the
> schema changes more often. Other services like keystone, heat, and
> ironic are considerably easier to work with and have well laid out
> tables that don't change much.
>
>
> FTR: Keystone also supports "no-downtime-upgrades" (just pending some
> functional tests before we apply for the tag) and we will be looking
> to move towards Alembic, so make sure that the code supplied can
> easily be swapped out between SQL-A-Migrate and Alembic (IIRC most
> projects want to move to alembic, but it is varying levels of
> difficulty to do so and therefore different priorities).
There are some things that I do like about Alembic and the way that it
heals the database. But there will probably be some tricky conversions
going from SQL Alchemy.
>
> I look forward to solid NDB support; having using NDB in the past to
> support another project, I always thought it could be an interesting
> choice to back OpenStack (++ to what Monty said eariler).
Thanks! I think the benefits will outweigh the investment for everyone.
>
> Thanks,
> Octave
>
>
> On 2/2/2017 1:25 PM, Mike Bayer wrote:
>>
>>
>> On 02/02/2017 02:52 PM, Mike Bayer wrote:
>>>
>>> But more critically I noticed you referred to altering the names of
>>> columns to suit NDB. How will this be accomplished? Changing
>>> a column
>>> name in an openstack application is no longer trivial, because
>>> online
>>> upgrades must be supported for applications like Nova and
>>> Neutron. A
>>> column name can't just change to a new name, both columns have
>>> to exist
>>> and logic must be added to keep these columns synchronized.
>>>
>>
>> correction, the phrase was "Row character length limits 65k ->
>> 14k" - does this refer to the total size of a row? I guess rows
>> that store JSON or tables like keystone tokens are what you had
>> in mind here, can you give specifics ?
>>
>>
>>
>> __________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> <mailto:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe>
>>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>
> --
>
> Oracle <http://www.oracle.com/>
> Octave J. Orgeron | Sr. Principal Architect and Software Engineer
> Oracle Linux OpenStack
> 500 Eldorado Blvd. | Broomfield, CO 80021
> Certified Oracle Enterprise Architect: Systems Infrastructure
> <http://www.oracle.com/us/solutions/enterprise-architecture/index.html>
> Green Oracle <http://www.oracle.com/commitment> Oracle is
> committed to developing practices and products that help protect
> the environment
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Oracle <http://www.oracle.com/>
Octave J. Orgeron | Sr. Principal Architect and Software Engineer
Oracle Linux OpenStack
<tel:+17206161550> 500 Eldorado Blvd. | Broomfield, CO 80021
Certified Oracle Enterprise Architect: Systems Infrastructure
<http://www.oracle.com/us/solutions/enterprise-architecture/index.html>
Green Oracle <http://www.oracle.com/commitment> Oracle is committed to
developing practices and products that help protect the environment
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170206/0ae9a27a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 658 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170206/0ae9a27a/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 2398 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170206/0ae9a27a/attachment.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 356 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170206/0ae9a27a/attachment-0001.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: oracle_sig_logo.gif
Type: image/gif
Size: 658 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170206/0ae9a27a/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: oea_logo.jpg
Type: image/jpeg
Size: 2398 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170206/0ae9a27a/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: green-for-email-sig_0.gif
Type: image/gif
Size: 356 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170206/0ae9a27a/attachment-0003.gif>
More information about the OpenStack-dev
mailing list