Open Stack

Excerpts from Steven Hardy's message of 2013-09-11 03:37:40 -0700:
> Hi all,
> 
> I'm investigating some issues, where data stored to a text column in mysql
> is silently truncated if it's too big.
> 
> It appears that the default configuration of mysql, and the sessions
> established via sqlalchemy is to simply warn on truncation rather than
> raise an error.
> 
> This seems to me to be almost never what you want, since on retrieval the
> data is corrupt and bad/unexpected stuff is likely.
> 
> This AFAICT is a mysql specific issue[1], which can be resolved by setting
> sql_mode to "traditional"[2,3], after which an error is raised on truncation,
> allowing us to catch the error before the data is stored.
> 
> My question is, how do other projects, or oslo.db, handle this atm?
> 
> It seems we either have to make sure the DB enforces the schema/model, or
> validate every single value before attempting to store, which seems like an
> unreasonable burden given that the schema changes pretty regularly.
> 
> Can any mysql, sqlalchemy and oslo.db experts pitch in with opinions on
> this?

I do think that setting stricter sql modes is the right way to go.

Note that I worked around this within Heat for JSON fields thusly:

https://git.openstack.org/cgit/openstack/heat/commit/?id=1e16ed2d

However, I do think we should make it a priority to protect the database
and the entire service from abnormally large values. The moment at which
we are serializing a data structure to the database is a bit late to
mitigate the cost of handling it. Here is an example of the kind of
border protection we need:

https://review.openstack.org/#/c/44585/

I want to detect that we overflowed a big column, and I think that if
it ever actually happens, it is a critical bug.