Open Stack

Hi Graham,

Comments below..

On 7/25/2017 5:04 AM, Graham Hayes wrote:
> On 24/07/17 20:37, Octave J. Orgeron wrote:
>
> <snip>
>
>>> Rather than having all the projects make use of
>>> oslo_db.sqlalchemy.ndb.AutoStringTinyText / AutoStringSize, we add new
>>> generic types to oslo.db :
>>>
>>> oslo_db.sqlalchemy.types.SmallString
>>> oslo_db.sqlalchemy.types.String
>>>
>>> (or similar )
>>>
>>> Internally, the ndb module would be mapping its implementation for
>>> AutoStringTinyText and AutoStringSize to these types.   Functionality
>>> would be identical, just the naming convention exported to downstream
>>> consuming projects would no longer refer to "ndb.<name>" for
>>> datatypes.
>> I think this would make sense.
>>
> <snip>
>> AutoStringSize, you pass two parameters, one being the non-NDB size and
>> the second being the NDB size. The point here is where you need to
>> reduce the size of the column to fit within the NDB limits, but you want
>> to preserve the String varchar type because it might be used in a key,
>> index, etc. I only use these in cases where the impacts are very low..
>> for example where a column is used for keeping track of status (up,
>> down, active, inactive, etc.) that don't require 255 varchars.
>>
>> In many cases, the use of these could be removed by simply changing the
>> columns to more appropriate types and sizes. There is a tremendous
>> amount of wasted space in many of the databases. I'm more than willing
>> to help out with this if teams decide they would rather do that instead
>> as the long-term solution. Until then, these functions enable the use of
>> both with minimal impact.
>>
>> Another thing to keep in mind is that the only services that I've had to
>> adjust column sizes for are:
>>
>> Cinder
>> Neutron
>> Nova
>> Magnum
>>
>> The other services that I'm working on like Keystone, Barbican, Murano,
>> Glance, etc. only need changes to:
>>
>> 1. Ensure that foreign keys are dropped and created in the correct order
>> when changing things like indexes, constraints, etc. Many services do
>> these proper steps already, there are just cases where this has been
>> missed because InnoDB is very forgiving on this. But other databases are
>> not.
>> 2. Fixing the database migration and sync operations to use oslo.db,
>> pass the right parameters, etc. Something that should have been done in
>> the first place, but hasn't. So this more of a house cleaning step to
>> insure that services are using oslo.db correctly.
>>
>> The only other oddball use case is deal with disabling nested
>> transactions, where Neutron is the only one that does this.
>>
>> On the flip side, here is a short list of services that I haven't had to
>> make ANY changes for other than having oslo.db 4.24 or above:
>>
>> aodh
>> gnocchi
>> heat
>> ironic
>> manila
> Which projects are you looking at?

If it's covered by the kolla framework, it's on the list :)

>
>>> 3. it's not clear (I don't even know right now by looking at these
>>> reviews) when one would use "AutoStringTinyText" or "AutoStringSize".
>>> For example in
>>> https://review.openstack.org/#/c/446643/10/nova/db/sqlalchemy/migrate_repo/versions/216_havana.py
>>>
>>> I see a list of String(255)'s changed to one type or the other without
>>> any clear notion why one would use one or the other.  Having names
>>> that define simply the declared nature of the type would be most
>>> appropriate.
>> One has to look at what the column is being used for and decide what
>> appropriate remediation steps are. This takes time and one must research
>> what kind of data goes in the column, what puts it there, what consumes
>> it, and what remediation would have the least amount of impact.
>>
> I have been out of the loop for a while - but I thought we were
> settling on one database, (MySQL over pgSQL) to ensure that we
> no longer had to have weird conditionals in our database layers and
> migrations?
>
> Is this something that someone is willing to commit to maintaining for
> all projects?
>
> I am just concerned we are adding in more custom code just as we are
> trying to indicate that we moving to MySQL (which I understand as MySQL
> like DB using an InnoDB based engine e.g. Maria, MySQL, Percona)[1]
>
> - Graham

MySQL Cluster is a MySQL database. It uses a different storage engine 
and clustering framework. You can read about the benefits and 
differences here:

https://review.openstack.org/#/c/429940/3/specs/mysql-cluster-support.rst

Oracle is committed to maintaining these patches, because our OpenStack 
distribution uses MySQL Cluster out of the box and has since version 2. 
Aside from services like Cinder, Neutron, and Nova, the impact of 
supporting MySQL Cluster is proving to be minimal. Even for Cinder, I 
only had to touch one table to make it work. The list of services that 
require nothing more than oslo.db 4.24 or above is increasing. Really 
the bigger issue that I'm seeing at this point are services that don't 
make proper use of oslo.db for their migrations.

So I want to make sure people understand that this isn't dumping a ton 
of custom code on every service. With the ndb module in oslo.db, a lot 
of the logic is abstracted. All that remains is to deal with are a few 
database tables, handling of some foreign keys, and in extreme cases 
like Neutron dealing with nested transactions. The rest of the work is 
making sure that oslo.db is being used for migrations, which I think 
will require more of an effort.



> 1 - Thinking about it - should https://review.openstack.org/#/c/427880
> refer to InnoDB vs just MySQL ?
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170725/36f1d785/attachment.html>