Open Stack

Hi Mike,

I've had a chance to look through the links you provided. I do think 
this is a rather heavy solution that would be more suited if there were 
actually significant dialect features to override from MySQL. MySQL and 
NDB use the same dialect and the differences really just come down to 
operation ordering, no support for savepoints, and no support for nested 
transactions. Even if you tried to do those operations today, SQL 
Alchemy is able to throw back appropriate errors telling you that you're 
doing something wrong or that the feature isn't supported. If we go down 
this path, we really only buy two things:

  * Ability to use the with_variant for setting column types.
  * Do some logic based on the selected dialect, which we would probably
    still have to set in oslo.db anyways as the hook.

It doesn't solve the issue of proper ordering of FK, constraints, or 
index operations. It doesn't remove the need to do variable 
substitutions where things are hard coded. And it doesn't resolve the 
issues where we have to intercept savepoints and nested transactions. It 
looks like the only major impact it would have is to reduce the number 
of if/then logic blocks in the SQL Alchemy and Alembic migration scripts.

But what does it cost to do this? Would the dialect be rolled into SQL 
Alchemy for the community, or would it be a separate plugin like 
Redshifts? Is it easier to maintain just the patches? Or would it mean 
more overhead for me to support the patches and the ndb dialect? I'd 
like to keep the overhead simple since it's just me at this point 
working on this.

So what I propose is that I'll update my patches for keystone and cinder 
next and post those for gerrit review. That will give folks a view into 
what the patches will look like and we can figure out if we want to 
change the approach. I'm also going to create a spec and blueprint to 
cover the changes across the services. I'll post links once all of that 
is up for review.

Thanks,
Octave

On 2/6/2017 7:53 AM, Mike Bayer wrote:
>
>
> On 02/03/2017 11:59 AM, Octave J. Orgeron wrote:
>> Hi Mike,
>>
>> Comments below..
>>
>> Thanks,
>> Octave
>>
>> On 2/3/2017 7:41 AM, Mike Bayer wrote:
>>>
>>>
>>> On 02/02/2017 05:28 PM, Octave J. Orgeron wrote:
>>>> That refers to the total length of the row. InnoDB has a limit of 65k
>>>> and NDB is limited to 14k.
>>>>
>>>> A simple example would be the volumes table in Cinder where the row
>>>> length goes beyond 14k. So in the IF logic block, I change columns 
>>>> types
>>>> that are vastly oversized such as status and attach_status, which by
>>>> default are 255 chars.
>>>
>>>
>>> let me give you a tip on IF blocks, that they are a bit of an
>>> anti-pattern.  If you want a column type to do one thing in one case,
>>> and another in another case, create an object that does the thing you
>>> want:
>>>
>>>
>>> some_table = Table(
>>>     'some_table', metadata,
>>>     Column('my_column', VARCHAR(255).with_variant(VARCHAR(50), 'ndb'))
>>> )
>>>
>>>
>>> I think we might want to look into creating a stub dialect called
>>> 'ndb' that subclasses mysql+pymysql.   Treating ndb as a whole
>>> different database means there's no longer the need for a flag in
>>> oslo.db, the 'ndb' name would instead be interpreted as a new backend
>>> - the main thing would be ensuring all the mysql-appropriate hooks in
>>> oslo.db are also emitted for ndb, but this also gives us a way to pick
>>> and choose which hooks apply.   It seems like there may be enough
>>> different about it to separate it at this level.
>>>
>>> Not sure if people on the list are seeing that we are simultaneously
>>> talking about getting rid of Postgresql in the efforts to support only
>>> "one database", while at the same time adding one that is in many ways
>>> a new database.
>>>
>>>
>>
>> This is an interesting approach as it would significantly reduce the
>> amount of code in my patches today. Do you have any pointers on where
>> this should be implemented as a stub? Would we have to take different
>> approaches for SQL Alchemy vs. Alembic?
>
> there are simple plugin points for both libraries.
>
> One of the popular 3rd party dialects right now is the 
> sqlalchemy-redshift dialect, which similarly to a lot of these 
> dialects is one that acts 95% like a "normal" dialect, in this case 
> postgresql, however various elements are overridden to provide 
> compatibility with Amazon's redshift.     The overlay of an NDB style 
> dialect on top of mysql would be a similar idea. The SQLAlchemy plugin 
> point consists of a setuptools entrypoint (see 
> https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/blob/master/setup.py#L40 
> , 
> https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/blob/master/sqlalchemy_redshift/dialect.py#L315) 
> and for Alembic, once the dialect is imported you define a special 
> Alembic class so that Alembic sees the engine name also (see 
> https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/blob/master/sqlalchemy_redshift/dialect.py#L19).
>
> In this case the NDB dialect seems like it may be a little bit of a 
> heavy solution but it would solve lots of issues like the 
> "mysql_engine" flag would no longer be in conflict, special datatypes 
> and naming schemes can be pulled in, etc.   It would at least allow 
> conditionals against "ndb" in Openstack projects to switch on the same 
> kind of criteria that they already do for sqlite/postgresql/mysql.
>
> It is possible for the ndb "stub dialect" to be at least temporarily 
> within oslo.db, however the way to go about this would be to start 
> getting ndb working as a proof of concept in terms of gerrit reviews. 
> that is, propose reviews to multiple projects and work at that level, 
> without actually merging anything.   We don't merge anything until 
> it's actually "done" as a tested and working feature / fix.
>
>
>
>
>>
>>
>> -- 
>>
>> Oracle <http://www.oracle.com/>
>> Octave J. Orgeron | Sr. Principal Architect and Software Engineer
>> Oracle Linux OpenStack
>> Mobile: +1-720-616-1550 <tel:+17206161550>
>> 500 Eldorado Blvd. | Broomfield, CO 80021
>> Certified Oracle Enterprise Architect: Systems Infrastructure
>> <http://www.oracle.com/us/solutions/enterprise-architecture/index.html>
>> Green Oracle <http://www.oracle.com/commitment> Oracle is committed to
>> developing practices and products that help protect the environment
>>
>>
>>
>> __________________________________________________________________________ 
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: 
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __________________________________________________________________________ 
>
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 

Oracle <http://www.oracle.com/>
Octave J. Orgeron | Sr. Principal Architect and Software Engineer
Oracle Linux OpenStack
Mobile: +1-720-616-1550 <tel:+17206161550>
500 Eldorado Blvd. | Broomfield, CO 80021
Certified Oracle Enterprise Architect: Systems Infrastructure 
<http://www.oracle.com/us/solutions/enterprise-architecture/index.html>
Green Oracle <http://www.oracle.com/commitment> Oracle is committed to 
developing practices and products that help protect the environment

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170206/832cd522/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: oracle_sig_logo.gif
Type: image/gif
Size: 658 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170206/832cd522/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: oea_logo.jpg
Type: image/jpeg
Size: 2398 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170206/832cd522/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: green-for-email-sig_0.gif
Type: image/gif
Size: 356 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170206/832cd522/attachment-0001.gif>