Open Stack

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 06/10/14 15:11, Jay Pipes wrote:
> On 10/06/2014 08:30 AM, Ihar Hrachyshka wrote: Hi all,
> 
> tl;dr we should enforce utf8 on server and client sides of db 
> connection, and this requires changes to docs and oslo.db. The
> latter may require raising effective libpq version dep to 9.1+.
> 
> Recently I was working on making sure we always run using utf8 on
> both database server and client side, and that we do it with all
> the proper connection options set (f.e. mysqldb requires
> use_unicode=0 to avoid performance drop - a thing that is not set
> anywhere in gate as of now, and not documented).
> 
> For server side, it's a matter of how we create databases, and in 
> devstack, it's usually utf8 (though, for unclear reason, latin1 is 
> still enforced for nova for initial creation due to Folsom (sic!)
> bug; I've reverted to utf8 with [1]).
> 
> For client side, it's not that clear. Though devstack enforces utf8
> in SQL connection strings, we don't ask to do so in official docs
> [2]. There is a bug [3] in LP that seems to be relevant.
> 
> I think the proper way is to enforce utf8 everywhere. The first
> step would be to make sure we suggest adding charset=utf8 to SQL
> connection strings, and configure SQL servers to use utf8. That is
> something to be covered by docs team.
> 
> But we can do better. We should also enforce utf8 on client side,
> so that there is no way to run with a different encoding, and so
> that we may get rid of additional options in sql connection
> strings. I've sent a patch for oslo.db [4] to do just that.
> 
>> So, the subject of this email is a bit misleading :) You are
>> suggesting a change to the UTF-8 charset enforcement here, and
>> below you are suggesting removing libpq<9.1 for related reasons.
> 
>> I suppose setting of utf8 on the client side, and supporting utf8
>> on the server side, but I do *not* recommend using a blanket
>> policy of charset=utf8 on every text column on the database side.
>> It really is wasteful for certain columns like uuids that are
>> used in joins all over the schema...

Maybe it is indeed wasteful, I don't have numbers; though the fact is
we don't allow any migrations for databases with any non utf8 tables
as of [1]. The code was copied in multiple projects (Nova, Glance
among other things). Also, the same check is present in oslo.db that
is used by most projects now.

Also we have migration rules in multiple projects that end up with
converting all tables to utf8. F.e. it's done in Nova [2].

So we already run against utf8 tables. Though we don't ever tell users
to create their databases with utf8 as default charset, opening a can
of worms. We also don't ever tell users to at least set use_unicode=0
when running against MySQLdb (which is the default and the only
supported MySQL driver as of Juno) to avoid significant performance
drop [3] (same is true for oursql driver, but that one is at least not
recommended to users thru official docs).

[1]: https://review.openstack.org/#/c/64764/
[2]:
https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/migrate_repo/versions/216_havana.py#L1539
[3]:
http://docs.sqlalchemy.org/en/rel_0_9/dialects/mysql.html#module-sqlalchemy.dialects.mysql.mysqldb

> 
>> Best, -jay
> 
> Though as you can see, the patch fails on py26 postgresql unit
> tests. This is because I set charset_encoding for PostgreSQL via
> libpq connection option that is available since PostgreSQL 9.1
> only. The failure is because we run py26 test suite on CentOS 6.5
> that still uses PostgreSQL 8.4.*, meaning the option is not
> available for us.
> 
> Now, there are several other ways available for us to enforce
> encoding without using the option. First, we may set encoding thru 
> PGCLIENTENCODING environment variable. Second, we may execute "SET 
> CLIENT_ENCODING TO utf8" after connecting to psql server.
> 
> Now, the question is whether it's worth reimplementing the feature 
> using some of the available hacks that will work with PSQL < 9.1. 
> We're about to drop Python 2.6 support in Kilo, and that probably 
> means that CentOS 6.5 will be dropped from gate too. Do we have
> any other distribution of interest that uses both python 2.7 and
> psql < 9.1, and is going to run Kilo pieces?
> 
> On Red Hat side, we're not interested in running Juno on EL6 that
> uses old psql. As for EL7, it has a psql version that supports the 
> connection option. On Ubuntu side, the only LTS release that is
> still supported and uses psql < 9.1 is Lucid, but it's based on
> py26, so it should not be an issue. Any other consumers that could
> be affected by effectively raising minimum version bar for psql to
> 9.1 in Kilo?
> 
> Though we've (almost) decided to drop py26 support in Kilo, we
> didn't have similar discussions for other components that we use.
> So I'd like to hear what's our take on that. Do we really drop
> *py26* support, or maybe what we really do is we drop *CentOS 6.5*
> support?..
> 
> On a more general note, do we track those kind of external 
> dependencies anywhere? There are lots of them: neutron depends on 
> specific versions of openvswitch; oslo.db depends on specific
> versions of underlying backend libraries, etc.
> 
> Comments?
> 
> [1]: https://review.openstack.org/126267 [2]: 
> http://docs.openstack.org/icehouse/install-guide/install/yum/content/nova-controller.html
>
>  [3]: https://bugs.launchpad.net/glance/+bug/1279000 [4]:
> https://review.openstack.org/111236
> 
> /Ihar
>> 
>> _______________________________________________ OpenStack-dev
>> mailing list OpenStack-dev at lists.openstack.org 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>> 
> _______________________________________________ OpenStack-dev
> mailing list OpenStack-dev at lists.openstack.org 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)

iQEcBAEBCgAGBQJUMp+pAAoJEC5aWaUY1u57yTgIAM/dKZtinHVDH8ZlO83y6tJe
SFzxDWlf57sQkHB1SY2mOzkTTTzjep8TbYjPpKNJAnZvITW2buCFWrj224Defqxl
wb+D71/kdN9PDh3ZqStTA7FRLXNTbwNJwIXK/UOKLGAVPiple7eiHalleGnHoedH
XHF5D3bDL9iVuBks/DzcBOf61tJpBANRocdCWEXfCmCVxCIV3PAdC5YFj09bukYR
cZpK+mBXmGDkfvYQctm7h/b7wu20Cxi2olj7Q2eDNx7sL5U6ar6RQaXAKxALeqBp
+QfzvjEvKC6hyvAyn36m4hoMhElQ7hOB1uzYmXQZbI0YzBYjjNID6sW2yWB8UK4=
=SQe1
-----END PGP SIGNATURE-----