Open Stack

On Jul 26, 2017 6:28 PM, "Michael Bayer" <mbayer at redhat.com> wrote:

On Wed, Jul 26, 2017 at 6:19 PM, Michael Bayer <mbayer at redhat.com> wrote:
> On Wed, Jul 26, 2017 at 5:30 PM, Michael Bayer <mbayer at redhat.com> wrote:
>>
>> There is a bigger problem with this entire series of changes, whether
>> or not the "ndb" keyword is present.  Which is that projects need to
>> add new columns, tables, and make datatype changes all the time, and
>> they will not have any idea about the requirements for ndb or even
>> that it exists, nor will anyone have access to this platform for
>> development nor should they be expected to worry about it.   If they
>> not only have to fill in dozens of special "ndb" or generic-but-needed
>> by ndb flags, and then if they even have to worry about the sum of all
>> the sizes in a row, that means the ndb implementation will be
>> continuously broken across many projects in every release unless ndb
>> developers are checking every database change in every project at all
>> times.   Is that level of effort part of the plan?
>
> OK, I apologize, you answered that here:
>
> https://review.openstack.org/#/c/427970/26
>
>
>> Now considering that my company is heavily invested in using MySQL
Cluster (NDB) and that we use the kola framework, we have to keep an eye on
each of the services to make sure that it works. This is why you see lots
of other patches that I'm working on to fix services like Cinder, Neutron,
Nova, etc. So has time goes by, we will continue to make patches to enable
these services to work with NDB.
>
> If we were to approach this as real migrations that change the lengths
> of datatypes, that means the tables must be created at InnoDB first
> and migrated to NDB within the migrations.  Because NDB will disallow
> the initial creation, right?
>
> if the proposal is to modify the actual sizes in the original
> migration files, that's not something that can be done, unfortunately,
> it would be hugely risky because those migrations represent a snapshot
> of the actual schema.
>
> If we *do* need to keep doing something like the "AutoStringXYZ"
> approach I really want to change those names and not have any "ndb."
> in it at all.

thinking out loud

oslo_db.sqlalchemy.types.String(255, mysql_small_rowsize=64)
oslo_db.sqlalchemy.types.String(255, mysql_small_rowsize=sa.TINYTEXT)
oslo_db.sqlalchemy.types.String(255, mysql_small_rowsize=sa.TEXT)


so if you don't have mysql_small_rowsize,  nothing happens.



Also, these flags in theory would *only* be in an old migration file.
Going forward, we'd hope that Oracle will be running its own CI and
ensuring new migrations don't go over the limits , right ?  New columns
would ideally not have conditional datatype rules at all.












>
> But all the options here seem kind of icky.
>
>
>
>
>
>
>>
>>
>>
>>
>>
>>
>>
>>>
>>> I don't see a way of automating that and making it maintainable without
a
>>> lot more overhead in code and people. If we really want to remove the
>>> complexity here, why don't we just change the sizes and types on these
>>> handful of table columns so that they fit within both InnoDB and NDB?
That
>>> way we don't need these functions and the tables are exactly the same?
That
>>> would only leave us with the createtable, savepoint/rollback, etc.
stuff to
>>> address which is already taken care of in the ndb module in oslo.db?
Then we
>>> just fix the foreign key stuff as I've been doing, since it has zero
impact
>>> on InnoDB deployments and if anything ensures things are consistent.
That
>>> would then leave us to really focus on fixing migrations to use oslo.db
and
>>> pass the correct flags, which is a more lengthy process than the rest of
>>> this.
>>>
>>> I don't see the point in trying to make this stuff anymore complicated.
>>>
>>> Octave
>>>
>>>
>>> On 7/25/2017 12:20 PM, Michael Bayer wrote:
>>>>
>>>> On Mon, Jul 24, 2017 at 5:41 PM, Michael Bayer <mbayer at redhat.com>
wrote:
>>>>>>
>>>>>> oslo_db.sqlalchemy.String(255, ndb_type=TINYTEXT) -> VARCHAR(255) for
>>>>>> most
>>>>>> dbs, TINYTEXT for ndb
>>>>>> oslo_db.sqlalchemy.String(4096, ndb_type=TEXT) -> VARCHAR(4096) for
most
>>>>>> dbs, TEXT for ndb
>>>>>> oslo_db.sqlalchemy.String(255, ndb_size=64) -> VARCHAR(255) on most
dbs,
>>>>>> VARCHAR(64) on ndb
>>>>>>
>>>>>> This way, we can override the String with TINYTEXT or TEXT or change
the
>>>>>> size for ndb.
>>>>>>>
>>>>>>> oslo_db.sqlalchemy.String(255)     -> VARCHAR(255) on most dbs,
>>>>>>> TINYTEXT() on ndb
>>>>>>> oslo_db.sqlalchemy.String(255, ndb_size=64)     -> VARCHAR(255) on
>>>>>>> most dbs, VARCHAR(64) on ndb
>>>>>>> oslo_db.sqlalchemy.String(50)     -> VARCHAR(50) on all dbs
>>>>>>> oslo_db.sqlalchemy.String(64)     -> VARCHAR(64) on all dbs
>>>>>>> oslo_db.sqlalchemy.String(80)     -> VARCHAR(64) on most dbs,
>>>>>>> TINYTEXT()
>>>>>>> on ndb
>>>>>>> oslo_db.sqlalchemy.String(80, ndb_size=55)     -> VARCHAR(64) on
most
>>>>>>> dbs, VARCHAR(55) on ndb
>>>>>>>
>>>>>>> don't worry about implementation, can the above declaration ->
>>>>>>> datatype mapping work ?
>>>>>>>
>>>>>>>
>>>>>> In my patch for Neutron, you'll see a lot of the AutoStringText()
calls
>>>>>> to
>>>>>> replace exceptionally long String columns (4096, 8192, and larger).
>>>>>
>>>>> MySQL supports large VARCHAR now, OK.   yeah this could be
>>>>> String(8192, ndb_type=TEXT) as well.
>>>>
>>>> OK, no, sorry each time I think of this I keep seeing the verbosity of
>>>> imports etc. in the code, because if we had:
>>>>
>>>> String(80, ndb_type=TEXT)
>>>>
>>>> then we have to import both String and TEXT, and then what if there's
>>>> ndb.TEXT, the code is still making an ndb-specific decision, etc.
>>>>
>>>> I still see that this can be mostly automated from a simple ruleset
>>>> based on the size:
>>>>
>>>> length <= 64 :    VARCHAR(length) on all backends
>>>> length > 64, length <= 255:   VARCHAR(length) for most backends,
>>>> TINYTEXT for ndb
>>>> length > 4096:  VARCHAR(length) for most backends, TEXT for ndb
>>>>
>>>> the one case that seems outside of this is:
>>>>
>>>> String(255)  where they have an index or key on the VARCHAR, and in
>>>> fact they only need < 64 characters to be indexed.  In that case you
>>>> don't want to use TINYTEXT, right?   So one exception:
>>>>
>>>> oslo_db.sqlalchemy.types.String(255, indexable=True)
>>>>
>>>> e.g. a declarative hint to the oslo_db backend to not use a LOB type.
>>>>
>>>> then we just need oslo_db.sqlalchemy.types.String, and virtually
>>>> nothing except the import has to change, and a few keywords.
>>>>
>>>> What we're trying to do in oslo_db is as much as possible state the
>>>> intent of a structure or datatype declaratively, and leave as much of
>>>> the implementation up to oslo_db itself.
>>>>
>>>> ____________________________________________________________
______________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:
unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>>
>>> ____________________________________________________________
______________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:
unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170726/871315c0/attachment-0001.html>