[openstack-dev] Sprint at Pycon: Port OpenStack to Python 3

John Dennis jdennis at redhat.com
Tue Apr 1 18:08:05 UTC 2014


On 04/01/2014 12:28 PM, Victor Stinner wrote:
> Le mardi 1 avril 2014, 09:44:11 John Dennis a écrit :
>>> The goal of the sprint is to port OpenStack components and
>>> OpenStack dependencies to Python 3,
>> 
>> This is a great goal, thank you! But I'm concerned it might be
>> premature.
> 
> The portage is already in progress. There are many components (8
> clients) where the py33 (Python 3.3) gate is voting. We try to keep
> this page up to date:
> 
> https://wiki.openstack.org/wiki/Python3
> 
> There are already a lot of dependencies which are already Python 3
> compatible, and the portage of OpenStack "server" components already
> started.
> 
>> My concern is this. The singled biggest change in Py2 -> Py3 is
>> string handling, especially with regards to str vs. unicode. We
>> have a significant number of bugs in the current code base with
>> regards to encoding exceptions, I just got done fixing a number of
>> them, I know there are others.
> 
> In which OpenStack component?

For one:

https://bugs.launchpad.net/keystone/+bug/1292311

But just looking at a lot of the OpenStack code it's easy to see things
are going to blow up once you start passing around non-ASCII characters.

> To be honest, we invest more time on fixing Python 3 issues than on
> adding new tests to check for non-regression. The problem is that
> currently, you cannot even import the Python module, so it's hard to
> run tests and harder to add new tests.


> I hope that it will become easier to run tests on Python 2 and Python
> 3, and to add more tests for non-ASCII data.

Yes, the fact the vast majority of the unit tests only pass ASCII values
is a significant problem.

Most of the problems are data driven. If you don't test with the data
that causes the problems you're not fully testing and that allows a lot
of problems to sneak through.

IMHO code reviews should not permit the inclusion of unit tests which do
not utilize test data containing non-ASCII characters. All existing unit
tests should be updated to use non-ASCII strings.

FWIW my last patch to one of the keystone LDAP unit test converted most
all the strings to contain non-ASCII characters. We should be doing this
for a lot of the test code.

> It's not easy to detect Unicode issues using Python 2 since most
> setup are in english, only no test using non-ASCII data right now,
> and Python 2 uses implicit conversion between bytes and Unicode
> strings.

See above.

It's not quite fair to say Py2 implicit conversions mask problems, in
many instances Py2's implicit conversions are the root cause of problems
(mainly because ASCII is the default encoding applied during the
implicit conversion).

> 
> It's much easier to detect Unicode issues using Python 3. I don't
> want to drop Python 2 support, just *add* Python 3 support. The code
> will work on 2.6-3.3.


-- 
John



More information about the OpenStack-dev mailing list