Open Stack

Excerpts from Christopher Yeoh's message of 2014-09-18 16:57:12 -0700:
> On Thu, 18 Sep 2014 12:12:28 -0400
> Sean Dague <sean at dague.net> wrote:
> > > When we can return the json-schema to user in the future, can we say
> > > that means API accepting utf8 or utf8mb4 is discoverable? If it is
> > > discoverable, then we needn't limit anything in our python code.
> > 
> > Honestly, we should accept utf8 (no weird mysqlism not quite utf8). We
> > should make the default scheme for our dbs support that on names (but
> > only for the name columns). The failure of a backend to do utf8 for
> > real should return an error to the user. Let's not make this more
> > complicated than it needs to be.
> 
> I agree that discoverability for this is not the way to go - I think its
> too complicated for end users. I don't know enough about mysql to know
> if utf8mb4 is going to a performance issue but if its not then we
> should just support utf-8 properly. 
> 
> We can we can catch the db errors. However whilst converting db
> errors causing 500s is fairly straightforward when an error occurs that
> deep in Nova it also means a lot of potential unwinding work in the db
> and compute layers which is complicated and error prone. So i'd prefer
> to avoid the situation with input validation in the first place. 

Just to add a reference into the discussion:

http://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html

It does have the same limitation of making fixed width keys and CHAR()
columns. It goes from 3 bytes per CHAR position, to 4, so it should not
be a database wide default, but something we use sparingly.

Note that the right answer for things that are not utf-8 (like UUID's)
is not to set a charset of latin1, but use BINARY/VARBINARY. Last
time I tried I had a difficult time coercing SQLAlchemy to model the
difference.. but maybe I just didn't look in the right part of the manual.