<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Mike,<br>
<br>
I've had a chance to look through the links you provided. I do think
this is a rather heavy solution that would be more suited if there
were actually significant dialect features to override from MySQL.
MySQL and NDB use the same dialect and the differences really just
come down to operation ordering, no support for savepoints, and no
support for nested transactions. Even if you tried to do those
operations today, SQL Alchemy is able to throw back appropriate
errors telling you that you're doing something wrong or that the
feature isn't supported. If we go down this path, we really only buy
two things:<br>
<ul>
<li>Ability to use the with_variant for setting column types.</li>
<li>Do some logic based on the selected dialect, which we would
probably still have to set in oslo.db anyways as the hook.<br>
</li>
</ul>
<p>It doesn't solve the issue of proper ordering of FK, constraints,
or index operations. It doesn't remove the need to do variable
substitutions where things are hard coded. And it doesn't resolve
the issues where we have to intercept savepoints and nested
transactions. It looks like the only major impact it would have is
to reduce the number of if/then logic blocks in the SQL Alchemy
and Alembic migration scripts.<br>
</p>
<p>But what does it cost to do this? Would the dialect be rolled
into SQL Alchemy for the community, or would it be a separate
plugin like Redshifts? Is it easier to maintain just the patches?
Or would it mean more overhead for me to support the patches and
the ndb dialect? I'd like to keep the overhead simple since it's
just me at this point working on this.<br>
</p>
So what I propose is that I'll update my patches for keystone and
cinder next and post those for gerrit review. That will give folks a
view into what the patches will look like and we can figure out if
we want to change the approach. I'm also going to create a spec and
blueprint to cover the changes across the services. I'll post links
once all of that is up for review.<br>
<br>
Thanks,<br>
Octave<br>
<br>
<div class="moz-cite-prefix">On 2/6/2017 7:53 AM, Mike Bayer wrote:<br>
</div>
<blockquote
cite="mid:7ee44d1c-6cd1-911d-bd30-6d9e3976b312@redhat.com"
type="cite">
<br>
<br>
On 02/03/2017 11:59 AM, Octave J. Orgeron wrote:
<br>
<blockquote type="cite">Hi Mike,
<br>
<br>
Comments below..
<br>
<br>
Thanks,
<br>
Octave
<br>
<br>
On 2/3/2017 7:41 AM, Mike Bayer wrote:
<br>
<blockquote type="cite">
<br>
<br>
On 02/02/2017 05:28 PM, Octave J. Orgeron wrote:
<br>
<blockquote type="cite">That refers to the total length of the
row. InnoDB has a limit of 65k
<br>
and NDB is limited to 14k.
<br>
<br>
A simple example would be the volumes table in Cinder where
the row
<br>
length goes beyond 14k. So in the IF logic block, I change
columns types
<br>
that are vastly oversized such as status and attach_status,
which by
<br>
default are 255 chars.
<br>
</blockquote>
<br>
<br>
let me give you a tip on IF blocks, that they are a bit of an
<br>
anti-pattern. If you want a column type to do one thing in
one case,
<br>
and another in another case, create an object that does the
thing you
<br>
want:
<br>
<br>
<br>
some_table = Table(
<br>
'some_table', metadata,
<br>
Column('my_column', VARCHAR(255).with_variant(VARCHAR(50),
'ndb'))
<br>
)
<br>
<br>
<br>
I think we might want to look into creating a stub dialect
called
<br>
'ndb' that subclasses mysql+pymysql. Treating ndb as a whole
<br>
different database means there's no longer the need for a flag
in
<br>
oslo.db, the 'ndb' name would instead be interpreted as a new
backend
<br>
- the main thing would be ensuring all the mysql-appropriate
hooks in
<br>
oslo.db are also emitted for ndb, but this also gives us a way
to pick
<br>
and choose which hooks apply. It seems like there may be
enough
<br>
different about it to separate it at this level.
<br>
<br>
Not sure if people on the list are seeing that we are
simultaneously
<br>
talking about getting rid of Postgresql in the efforts to
support only
<br>
"one database", while at the same time adding one that is in
many ways
<br>
a new database.
<br>
<br>
<br>
</blockquote>
<br>
This is an interesting approach as it would significantly reduce
the
<br>
amount of code in my patches today. Do you have any pointers on
where
<br>
this should be implemented as a stub? Would we have to take
different
<br>
approaches for SQL Alchemy vs. Alembic?
<br>
</blockquote>
<br>
there are simple plugin points for both libraries.
<br>
<br>
One of the popular 3rd party dialects right now is the
sqlalchemy-redshift dialect, which similarly to a lot of these
dialects is one that acts 95% like a "normal" dialect, in this
case postgresql, however various elements are overridden to
provide compatibility with Amazon's redshift. The overlay of
an NDB style dialect on top of mysql would be a similar idea.
The SQLAlchemy plugin point consists of a setuptools entrypoint
(see
<a class="moz-txt-link-freetext" href="https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/blob/master/setup.py#L40">https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/blob/master/setup.py#L40</a>
,
<a class="moz-txt-link-freetext" href="https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/blob/master/sqlalchemy_redshift/dialect.py#L315">https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/blob/master/sqlalchemy_redshift/dialect.py#L315</a>)
and for Alembic, once the dialect is imported you define a special
Alembic class so that Alembic sees the engine name also (see
<a class="moz-txt-link-freetext" href="https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/blob/master/sqlalchemy_redshift/dialect.py#L19">https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/blob/master/sqlalchemy_redshift/dialect.py#L19</a>).<br>
<br>
In this case the NDB dialect seems like it may be a little bit of
a heavy solution but it would solve lots of issues like the
"mysql_engine" flag would no longer be in conflict, special
datatypes and naming schemes can be pulled in, etc. It would at
least allow conditionals against "ndb" in Openstack projects to
switch on the same kind of criteria that they already do for
sqlite/postgresql/mysql.
<br>
<br>
It is possible for the ndb "stub dialect" to be at least
temporarily within oslo.db, however the way to go about this would
be to start getting ndb working as a proof of concept in terms of
gerrit reviews. that is, propose reviews to multiple projects and
work at that level, without actually merging anything. We don't
merge anything until it's actually "done" as a tested and working
feature / fix.
<br>
<br>
<br>
<br>
<br>
<blockquote type="cite">
<br>
<br>
--
<br>
<br>
Oracle <a class="moz-txt-link-rfc2396E" href="http://www.oracle.com/"><http://www.oracle.com/></a>
<br>
Octave J. Orgeron | Sr. Principal Architect and Software
Engineer
<br>
Oracle Linux OpenStack
<br>
Mobile: +1-720-616-1550 <tel:+17206161550>
<br>
500 Eldorado Blvd. | Broomfield, CO 80021
<br>
Certified Oracle Enterprise Architect: Systems Infrastructure
<br>
<a class="moz-txt-link-rfc2396E" href="http://www.oracle.com/us/solutions/enterprise-architecture/index.html"><http://www.oracle.com/us/solutions/enterprise-architecture/index.html></a>
<br>
Green Oracle <a class="moz-txt-link-rfc2396E" href="http://www.oracle.com/commitment"><http://www.oracle.com/commitment></a> Oracle is
committed to
<br>
developing practices and products that help protect the
environment
<br>
<br>
<br>
<br>
__________________________________________________________________________
<br>
OpenStack Development Mailing List (not for usage questions)
<br>
Unsubscribe:
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<br>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
<br>
<br>
</blockquote>
<br>
__________________________________________________________________________
<br>
OpenStack Development Mailing List (not for usage questions)
<br>
Unsubscribe:
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<br>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
<br>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<br>
<p>
<a href="http://www.oracle.com/" target="_blank"><img
src="cid:part1.07040809.00040106@oracle.com" alt="Oracle"
border="0" height="26" width="114"></a><br>
<font color="#666666" size="2" face="Verdana, Arial, Helvetica,
sans-serif">Octave J. Orgeron | Sr. Principal Architect and
Software Engineer<br>
<font color="#ff0000">Oracle</font> Linux OpenStack<br>
Mobile: <a href="tel:+17206161550">+1-720-616-1550</a><br>
500 Eldorado Blvd. | Broomfield, CO 80021<br>
<a
href="http://www.oracle.com/us/solutions/enterprise-architecture/index.html"><img
src="cid:part4.01030001.06070904@oracle.com"
alt="Certified Oracle Enterprise Architect: Systems
Infrastructure" border="0" height="42" width="182"></a><br>
<a href="http://www.oracle.com/commitment" target="_blank"><img
src="cid:part6.08050804.00010009@oracle.com" alt="Green
Oracle" align="absmiddle" border="0" height="28"
width="44"></a>
<font color="#4B7D42" size="1" face="Verdana, Arial,
Helvetica, sans-serif">Oracle is committed to developing
practices and products that help protect the environment</font>
</font></p>
<font color="#666666" size="2" face="Verdana, Arial, Helvetica,
sans-serif">
</font></div>
</body>
</html>