Open Stack

On 10/06/2014 08:30 AM, Ihar Hrachyshka wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> Hi all,
>
> tl;dr we should enforce utf8 on server and client sides of db
> connection, and this requires changes to docs and oslo.db. The latter
> may require raising effective libpq version dep to 9.1+.
>
> Recently I was working on making sure we always run using utf8 on both
> database server and client side, and that we do it with all the proper
> connection options set (f.e. mysqldb requires use_unicode=0 to avoid
> performance drop - a thing that is not set anywhere in gate as of now,
> and not documented).
>
> For server side, it's a matter of how we create databases, and in
> devstack, it's usually utf8 (though, for unclear reason, latin1 is
> still enforced for nova for initial creation due to Folsom (sic!) bug;
> I've reverted to utf8 with [1]).
>
> For client side, it's not that clear. Though devstack enforces utf8 in
> SQL connection strings, we don't ask to do so in official docs [2].
> There is a bug [3] in LP that seems to be relevant.
>
> I think the proper way is to enforce utf8 everywhere. The first step
> would be to make sure we suggest adding charset=utf8 to SQL connection
> strings, and configure SQL servers to use utf8. That is something to
> be covered by docs team.
>
> But we can do better. We should also enforce utf8 on client side, so
> that there is no way to run with a different encoding, and so that we
> may get rid of additional options in sql connection strings. I've sent
> a patch for oslo.db [4] to do just that.

So, the subject of this email is a bit misleading :) You are suggesting 
a change to the UTF-8 charset enforcement here, and below you are 
suggesting removing libpq<9.1 for related reasons.

I suppose setting of utf8 on the client side, and supporting utf8 on the 
server side, but I do *not* recommend using a blanket policy of 
charset=utf8 on every text column on the database side. It really is 
wasteful for certain columns like uuids that are used in joins all over 
the schema...

Best,
-jay

> Though as you can see, the patch fails on py26 postgresql unit tests.
> This is because I set charset_encoding for PostgreSQL via libpq
> connection option that is available since PostgreSQL 9.1 only. The
> failure is because we run py26 test suite on CentOS 6.5 that still
> uses PostgreSQL 8.4.*, meaning the option is not available for us.
>
> Now, there are several other ways available for us to enforce encoding
> without using the option. First, we may set encoding thru
> PGCLIENTENCODING environment variable. Second, we may execute "SET
> CLIENT_ENCODING TO utf8" after connecting to psql server.
>
> Now, the question is whether it's worth reimplementing the feature
> using some of the available hacks that will work with PSQL < 9.1.
> We're about to drop Python 2.6 support in Kilo, and that probably
> means that CentOS 6.5 will be dropped from gate too. Do we have any
> other distribution of interest that uses both python 2.7 and psql <
> 9.1, and is going to run Kilo pieces?
>
> On Red Hat side, we're not interested in running Juno on EL6 that uses
> old psql. As for EL7, it has a psql version that supports the
> connection option. On Ubuntu side, the only LTS release that is still
> supported and uses psql < 9.1 is Lucid, but it's based on py26, so it
> should not be an issue. Any other consumers that could be affected by
> effectively raising minimum version bar for psql to 9.1 in Kilo?
>
> Though we've (almost) decided to drop py26 support in Kilo, we didn't
> have similar discussions for other components that we use. So I'd like
> to hear what's our take on that. Do we really drop *py26* support, or
> maybe what we really do is we drop *CentOS 6.5* support?..
>
> On a more general note, do we track those kind of external
> dependencies anywhere? There are lots of them: neutron depends on
> specific versions of openvswitch; oslo.db depends on specific versions
> of underlying backend libraries, etc.
>
> Comments?
>
> [1]: https://review.openstack.org/126267
> [2]:
> http://docs.openstack.org/icehouse/install-guide/install/yum/content/nova-controller.html
> [3]: https://bugs.launchpad.net/glance/+bug/1279000
> [4]: https://review.openstack.org/111236
>
> /Ihar
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
>
> iQEcBAEBCgAGBQJUMotwAAoJEC5aWaUY1u57IQQIAN2yHfxbj0DYDMHE40OGbEwU
> 8fis/KOAxSyUEhHbbECS8ODGjHNK9SeuddUplzL5sKVbYIMZgt3wKXGT/rT5ex0C
> iMmc7/mb6JGGTUYvR1sF2EXxioJ3kWJSt5oQqBhVT14y2g8rwqN6tQaTusfDVZWN
> yZ7KCmQDc+3HMyY3EmYCp6V1ndubLbx8Oh56/jtdSGdNtc7V4IwYIF3F4WQaKBcB
> 54tZkAO0wvlbbRrXqEiyf99/dVLDiMquM3LNKEbYg4zjDUEX74fGJ3mscQtWdDLv
> 0TcqOvQ2pwLvpksf30kvBBHGcNCF9x8Ov8H4vg5wRTD4bkpjEDwmXzQXTk7xfHM=
> =aYRM
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>