<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Graham,<br>
<br>
Comments below..<br>
<br>
<div class="moz-cite-prefix">On 7/25/2017 5:04 AM, Graham Hayes
wrote:<br>
</div>
<blockquote cite="mid:b24031da-191d-13c3-2816-5dfb19bbe52f@suse.de"
type="cite">
<pre wrap="">On 24/07/17 20:37, Octave J. Orgeron wrote:
<snip>
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
Rather than having all the projects make use of
oslo_db.sqlalchemy.ndb.AutoStringTinyText / AutoStringSize, we add new
generic types to oslo.db :
oslo_db.sqlalchemy.types.SmallString
oslo_db.sqlalchemy.types.String
(or similar )
Internally, the ndb module would be mapping its implementation for
AutoStringTinyText and AutoStringSize to these types. Functionality
would be identical, just the naming convention exported to downstream
consuming projects would no longer refer to "ndb.<name>" for
datatypes.
</pre>
</blockquote>
<pre wrap="">
I think this would make sense.
</pre>
<blockquote type="cite">
<pre wrap="">
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<pre wrap=""><snip>
</pre>
<blockquote type="cite">
<pre wrap="">
AutoStringSize, you pass two parameters, one being the non-NDB size and
the second being the NDB size. The point here is where you need to
reduce the size of the column to fit within the NDB limits, but you want
to preserve the String varchar type because it might be used in a key,
index, etc. I only use these in cases where the impacts are very low..
for example where a column is used for keeping track of status (up,
down, active, inactive, etc.) that don't require 255 varchars.
In many cases, the use of these could be removed by simply changing the
columns to more appropriate types and sizes. There is a tremendous
amount of wasted space in many of the databases. I'm more than willing
to help out with this if teams decide they would rather do that instead
as the long-term solution. Until then, these functions enable the use of
both with minimal impact.
Another thing to keep in mind is that the only services that I've had to
adjust column sizes for are:
Cinder
Neutron
Nova
Magnum
The other services that I'm working on like Keystone, Barbican, Murano,
Glance, etc. only need changes to:
1. Ensure that foreign keys are dropped and created in the correct order
when changing things like indexes, constraints, etc. Many services do
these proper steps already, there are just cases where this has been
missed because InnoDB is very forgiving on this. But other databases are
not.
2. Fixing the database migration and sync operations to use oslo.db,
pass the right parameters, etc. Something that should have been done in
the first place, but hasn't. So this more of a house cleaning step to
insure that services are using oslo.db correctly.
The only other oddball use case is deal with disabling nested
transactions, where Neutron is the only one that does this.
On the flip side, here is a short list of services that I haven't had to
make ANY changes for other than having oslo.db 4.24 or above:
aodh
gnocchi
heat
ironic
manila
</pre>
</blockquote>
<pre wrap="">
Which projects are you looking at?</pre>
</blockquote>
<br>
If it's covered by the kolla framework, it's on the list :)<br>
<br>
<blockquote cite="mid:b24031da-191d-13c3-2816-5dfb19bbe52f@suse.de"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
3. it's not clear (I don't even know right now by looking at these
reviews) when one would use "AutoStringTinyText" or "AutoStringSize".
For example in
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/446643/10/nova/db/sqlalchemy/migrate_repo/versions/216_havana.py">https://review.openstack.org/#/c/446643/10/nova/db/sqlalchemy/migrate_repo/versions/216_havana.py</a>
I see a list of String(255)'s changed to one type or the other without
any clear notion why one would use one or the other. Having names
that define simply the declared nature of the type would be most
appropriate.
</pre>
</blockquote>
<pre wrap="">
One has to look at what the column is being used for and decide what
appropriate remediation steps are. This takes time and one must research
what kind of data goes in the column, what puts it there, what consumes
it, and what remediation would have the least amount of impact.
</pre>
</blockquote>
<pre wrap="">
I have been out of the loop for a while - but I thought we were
settling on one database, (MySQL over pgSQL) to ensure that we
no longer had to have weird conditionals in our database layers and
migrations?
Is this something that someone is willing to commit to maintaining for
all projects?
I am just concerned we are adding in more custom code just as we are
trying to indicate that we moving to MySQL (which I understand as MySQL
like DB using an InnoDB based engine e.g. Maria, MySQL, Percona)[1]
- Graham
</pre>
</blockquote>
<br>
MySQL Cluster is a MySQL database. It uses a different storage
engine and clustering framework. You can read about the benefits and
differences here:<br>
<br>
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/429940/3/specs/mysql-cluster-support.rst">https://review.openstack.org/#/c/429940/3/specs/mysql-cluster-support.rst</a><br>
<br>
Oracle is committed to maintaining these patches, because our
OpenStack distribution uses MySQL Cluster out of the box and has
since version 2. Aside from services like Cinder, Neutron, and Nova,
the impact of supporting MySQL Cluster is proving to be minimal.
Even for Cinder, I only had to touch one table to make it work. The
list of services that require nothing more than oslo.db 4.24 or
above is increasing. Really the bigger issue that I'm seeing at this
point are services that don't make proper use of oslo.db for their
migrations.<br>
<br>
So I want to make sure people understand that this isn't dumping a
ton of custom code on every service. With the ndb module in oslo.db,
a lot of the logic is abstracted. All that remains is to deal with
are a few database tables, handling of some foreign keys, and in
extreme cases like Neutron dealing with nested transactions. The
rest of the work is making sure that oslo.db is being used for
migrations, which I think will require more of an effort.<br>
<br>
<br>
<br>
<blockquote cite="mid:b24031da-191d-13c3-2816-5dfb19bbe52f@suse.de"
type="cite">
<pre wrap="">
1 - Thinking about it - should <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/427880">https://review.openstack.org/#/c/427880</a>
refer to InnoDB vs just MySQL ?</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
<div class="moz-signature"><br>
<font color="#666666" size="2" face="Verdana, Arial, Helvetica,
sans-serif"><font color="#666666" size="2" face="Verdana, Arial,
Helvetica, sans-serif"></font></font></div>
</body>
</html>