<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Comments below..<br>
<br>
<div class="moz-cite-prefix">On 2/3/2017 8:34 AM, Mike Bayer wrote:<br>
</div>
<blockquote
cite="mid:0ec5e54b-ac92-c00a-412b-dfc68933d99e@redhat.com"
type="cite">
<br>
<br>
On 02/03/2017 10:21 AM, Doug Hellmann wrote:
<br>
<blockquote type="cite">Excerpts from Mike Bayer's message of
2017-02-03 09:41:11 -0500:
<br>
<blockquote type="cite">
<br>
On 02/02/2017 05:28 PM, Octave J. Orgeron wrote:
<br>
<blockquote type="cite">That refers to the total length of the
row. InnoDB has a limit of 65k
<br>
and NDB is limited to 14k.
<br>
<br>
A simple example would be the volumes table in Cinder where
the row
<br>
length goes beyond 14k. So in the IF logic block, I change
columns types
<br>
that are vastly oversized such as status and attach_status,
which by
<br>
default are 255 chars.
<br>
</blockquote>
<br>
<br>
let me give you a tip on IF blocks, that they are a bit of an
<br>
anti-pattern. If you want a column type to do one thing in
one case,
<br>
and another in another case, create an object that does the
thing you want:
<br>
<br>
<br>
some_table = Table(
<br>
'some_table', metadata,
<br>
Column('my_column',
VARCHAR(255).with_variant(VARCHAR(50), 'ndb'))
<br>
)
<br>
</blockquote>
<br>
I wonder if we want to do either, though. Shouldn't we try to
use
<br>
the same (smaller) column size all the time? Otherwise we end up
<br>
with another incompatibility between different deployments,
since
<br>
sometimes things like names might have different sizes in
different
<br>
clouds.
<br>
</blockquote>
<br>
in that case you have to do a migration which as you know these
days means the "old" column remains for a whole release cycle and
the application must undergo significant complexity, either at the
app level or in triggers, to keep data between "old" and "new"
columns simultaneously. So one advantage to keeping this at the
"create for NDB" level is that we don't need to get into schema
migrations.
<br>
<br>
Unless we changed the value in the application and its migration
files completely, and *didnt* migrate old applications, and just
hope/ensure that they aren't writing larger data values. Maybe
that's possible though it seems a little scary. Perhaps some
kind of annotated type like VARCHAR(50, unmigrated=255) to note
what's going on.
<br>
<br>
<br>
</blockquote>
<br>
This is one of the things that I worried about and why I took the
approach of doing the logic for NDB and keeping the default logic
there based on the mysql_storage_engine setting. Perhaps it makes
sense to do things this way first and then in a future release do
the migrations of the column sizes and types as a second phase. Then
as a third phase we can remove the column size and type logic
changes? At that point, the only real change left will be the
substitution of the "mysql_engine=InnoDB" with a value that we could
abstract from somewhere (oslo.db or a dialect stub).<br>
<br>
As for the foreign key, constraints, and index ordering, it's good
practice to do things in the right order and doesn't impact any
functionality in InnoDB. So that's actually a plus that I'll fix
those with my patches.<br>
<br>
Then that will just leave the logic for dealing with savepoints and
nested operations. Then when NDB adds those features, we can drop
that logic down the road.<br>
<br>
<blockquote
cite="mid:0ec5e54b-ac92-c00a-412b-dfc68933d99e@redhat.com"
type="cite">
<br>
<br>
<blockquote type="cite">
<br>
<blockquote type="cite">I think we might want to look into
creating a stub dialect called 'ndb'
<br>
that subclasses mysql+pymysql. Treating ndb as a whole
different
<br>
database means there's no longer the need for a flag in
oslo.db, the
<br>
'ndb' name would instead be interpreted as a new backend - the
main
<br>
thing would be ensuring all the mysql-appropriate hooks in
oslo.db are
<br>
also emitted for ndb, but this also gives us a way to pick and
choose
<br>
which hooks apply. It seems like there may be enough
different about
<br>
it to separate it at this level.
<br>
<br>
Not sure if people on the list are seeing that we are
simultaneously
<br>
talking about getting rid of Postgresql in the efforts to
support only
<br>
"one database", while at the same time adding one that is in
many ways a
<br>
new database.
<br>
</blockquote>
<br>
Yes, that does seem a bit ironic. That's also why I was pointing
<br>
out that we're going to want to have people lined up to support
the
<br>
work before starting. The lack of help with Postresql testing
<br>
resulted in removing it from the gate, and possibly to dropping
<br>
support entirely.
<br>
<br>
For reference, the discussion in [1] led to this proposed TC
<br>
resolution [2].
<br>
<br>
[1]
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/pipermail/openstack-dev/2017-February/thread.html#111357">http://lists.openstack.org/pipermail/openstack-dev/2017-February/thread.html#111357</a><br>
[2] <a class="moz-txt-link-freetext" href="https://review.openstack.org/427880">https://review.openstack.org/427880</a>
<br>
<br>
<blockquote type="cite">
<br>
<br>
<br>
<br>
So to determine a more appropriate size, I look
<br>
<blockquote type="cite">through the Cinder code to find where
the possible options/states are
<br>
for those columns. Then I cut it down to a more reasonable
size. I'm
<br>
very careful when I cut the size of a string column to
ensure that all
<br>
of the possible values can be contained.
<br>
<br>
In cases where a column is extremely large for capturing the
outputs of
<br>
a command, I will change the type to Text or TinyText
depending on the
<br>
length required. A good example of this is in the agents
table of
<br>
Neutron where there is a column for configurations that has
a string
<br>
length of 4096 characters, which I change to Text. Text
blobs are stored
<br>
differently and do not count against the row length.
<br>
<br>
I've also observed differences between Kilo, Mitaka, and tip
where even
<br>
for InnoDB some of these tables are getting wider than can
be supported.
<br>
So in the case of Cinder, some of the columns have been
shifted to
<br>
separate tables to fit within 65k. I've seen the same thing
in Neutron.
<br>
So I fully expect that some of the services that have table
bloat will
<br>
have to cut the lengths or break the tables up over time
anyways. As
<br>
that happens, it reduces the amount of work for me, which is
a good thing.
<br>
<br>
The most complicated database schemas to patch up are
cinder, glance,
<br>
neutron, and nova due to the size and complexity of their
tables. Those
<br>
also have a lot of churn between releases where the schema
changes more
<br>
often. Other services like keystone, heat, and ironic are
considerably
<br>
easier to work with and have well laid out tables that don't
change much.
<br>
<br>
Thanks,
<br>
Octave
<br>
<br>
On 2/2/2017 1:25 PM, Mike Bayer wrote:
<br>
<blockquote type="cite">
<br>
<br>
On 02/02/2017 02:52 PM, Mike Bayer wrote:
<br>
<blockquote type="cite">
<br>
But more critically I noticed you referred to altering
the names of
<br>
columns to suit NDB. How will this be accomplished?
Changing a column
<br>
name in an openstack application is no longer trivial,
because online
<br>
upgrades must be supported for applications like Nova
and Neutron. A
<br>
column name can't just change to a new name, both
columns have to exist
<br>
and logic must be added to keep these columns
synchronized.
<br>
<br>
</blockquote>
<br>
correction, the phrase was "Row character length limits
65k -> 14k" -
<br>
does this refer to the total size of a row? I guess rows
that store
<br>
JSON or tables like keystone tokens are what you had in
mind here, can
<br>
you give specifics ?
<br>
<br>
<br>
<br>
__________________________________________________________________________
<br>
<br>
OpenStack Development Mailing List (not for usage
questions)
<br>
Unsubscribe:
<br>
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<br>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
<br>
<br>
</blockquote>
<br>
--
<br>
<br>
Oracle <a class="moz-txt-link-rfc2396E" href="http://www.oracle.com/"><http://www.oracle.com/></a>
<br>
Octave J. Orgeron | Sr. Principal Architect and Software
Engineer
<br>
Oracle Linux OpenStack
<br>
Mobile: +1-720-616-1550 <tel:+17206161550>
<br>
500 Eldorado Blvd. | Broomfield, CO 80021
<br>
Certified Oracle Enterprise Architect: Systems
Infrastructure
<br>
<a class="moz-txt-link-rfc2396E" href="http://www.oracle.com/us/solutions/enterprise-architecture/index.html"><http://www.oracle.com/us/solutions/enterprise-architecture/index.html></a>
<br>
Green Oracle <a class="moz-txt-link-rfc2396E" href="http://www.oracle.com/commitment"><http://www.oracle.com/commitment></a> Oracle
is committed to
<br>
developing practices and products that help protect the
environment
<br>
<br>
<br>
<br>
__________________________________________________________________________
<br>
OpenStack Development Mailing List (not for usage questions)
<br>
Unsubscribe:
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<br>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
<br>
<br>
</blockquote>
<br>
</blockquote>
<br>
__________________________________________________________________________
<br>
OpenStack Development Mailing List (not for usage questions)
<br>
Unsubscribe:
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<br>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
<br>
<br>
</blockquote>
<br>
__________________________________________________________________________
<br>
OpenStack Development Mailing List (not for usage questions)
<br>
Unsubscribe:
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<br>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
<br>
<br>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<br>
<p>
<a href="http://www.oracle.com/" target="_blank"><img
src="cid:part1.06000309.09020800@oracle.com" alt="Oracle"
border="0" height="26" width="114"></a><br>
<font color="#666666" size="2" face="Verdana, Arial, Helvetica,
sans-serif">Octave J. Orgeron | Sr. Principal Architect and
Software Engineer<br>
<font color="#ff0000">Oracle</font> Linux OpenStack<br>
Mobile: <a href="tel:+17206161550">+1-720-616-1550</a><br>
500 Eldorado Blvd. | Broomfield, CO 80021<br>
<a
href="http://www.oracle.com/us/solutions/enterprise-architecture/index.html"><img
src="cid:part4.00010806.07040207@oracle.com"
alt="Certified Oracle Enterprise Architect: Systems
Infrastructure" border="0" height="42" width="182"></a><br>
<a href="http://www.oracle.com/commitment" target="_blank"><img
src="cid:part6.03010108.09040308@oracle.com" alt="Green
Oracle" align="absmiddle" border="0" height="28"
width="44"></a>
<font color="#4B7D42" size="1" face="Verdana, Arial,
Helvetica, sans-serif">Oracle is committed to developing
practices and products that help protect the environment</font>
</font></p>
<font color="#666666" size="2" face="Verdana, Arial, Helvetica,
sans-serif">
</font></div>
</body>
</html>