Open Stack

Le mardi 1 avril 2014, 09:44:11 John Dennis a écrit :
> > The goal of the sprint is to port OpenStack components and OpenStack
> > dependencies to Python 3,
> 
> This is a great goal, thank you! But I'm concerned it might be premature.

The portage is already in progress. There are many components (8 clients) 
where the py33 (Python 3.3) gate is voting. We try to keep this page up to 
date:

   https://wiki.openstack.org/wiki/Python3

There are already a lot of dependencies which are already Python 3 compatible, 
and the portage of OpenStack "server" components already started.

> My concern is this. The singled biggest change in Py2 -> Py3 is string
> handling, especially with regards to str vs. unicode. We have a
> significant number of bugs in the current code base with regards to
> encoding exceptions, I just got done fixing a number of them, I know
> there are others.

In which OpenStack component?

> While I was fixing them I searched the OpenStack
> coding guidelines to find out coding practices were supposed to be
> enforcing with regards to non-ASCII strings and discovered there is
> isn't much, it seems incomplete. Some of it seems based more on
> speculation than actual knowledge of defined Python behavior. I'm not
> sure, but given we do not have clear guidelines for unicode in Py2,
> never mind guidelines that will allow running under both Py2 and Py3 I'm
> willing to guess we have little in the gate testing that enforces any
> string handling guidelines.

It's an ongoing effort. We are slowly porting code but also adding new tests 
for non-ASCII data. For example, one of my recent patch for swiftclient adds 
new tests for non-ASCII URLs, whereas the existing tests only test ASCII 
(which is irrevelant for this specific test):

   https://review.openstack.org/#/c/84102/

To be honest, we invest more time on fixing Python 3 issues than on adding new 
tests to check for non-regression. The problem is that currently, you cannot 
even import the Python module, so it's hard to run tests and harder to add new 
tests.

I hope that it will become easier to run tests on Python 2 and Python 3, and 
to add more tests for non-ASCII data.

> I'm just in the process of finishing up a document to address these
> concerns. Unfortunately I'm going to be off-line for several weeks and I
> didn't want to start a discussion I couldn't participate in (plus there
> are some Py3 issues in the document I need to clean up) so I was going
> to wait to post it.

There are some "guidelines" to port Python 2 code on Python 3 on the wiki 
page. You may complete it?

   https://wiki.openstack.org/wiki/Python3

> My concern is we need to get our Py2 house in order *before* tackling
> Py3 porting. Doing Py3 porting before we have clear guidelines on
> unicode, str, bytes, encoding, etc. along with gate tests that enforce
> these guidelines is putting the cart before the horse. Whatever patches
> come out of a Py3 porting sprint might have to be completely redone.

It's not easy to detect Unicode issues using Python 2 since most setup are in 
english, only no test using non-ASCII data right now, and Python 2 uses 
implicit conversion between bytes and Unicode strings.

It's much easier to detect Unicode issues using Python 3. I don't want to drop 
Python 2 support, just *add* Python 3 support. The code will work on 2.6-3.3.

> FWIW projects that deal with web services, wire protocols, external
> datastores, etc. who have already started porting to Py3 have
> encountered significant pain points with Py3, some of which is just
> being resolved and which have caused on-going changes in Py3. We deal
> with a lot of these same issues in OpenStack. Before we just start
> hacking away I think it would behoove us to first have a very clear and
> explicit document on how we're going to address these issues *before* we
> start changing code.

There are many web servers and clients already running on Python 3. For 
example, Django supports Python 3 since its version 1.5 (released in February 
2013). I ported Paste and PasteScript on Python 3, but there are also web 
modules already Python 3 compatible (ex: WebOb).

Victor