Open Stack

On Thu, Oct 17, 2013 at 2:24 PM, John Dennis <jdennis at redhat.com> wrote:

> On 10/17/2013 12:22 PM,  Luis A. Garcia wrote:
> > On 10/16/2013 1:11 PM, Doug Hellmann wrote:
> >>
> >> [snip]
> >> Option 3 is closer to the new plan for Icehouse, which is to have _()
> >> return a Message, allow Message to work in a few contexts like a string
> >> (so that, for example, log calls and exceptions can be left alone, even
> >> if they use % to combine a translated string with arguments), but then
> >> have the logging and API code explicitly handle the translation of
> >> Message instances so we can always pass unicode objects outside of
> >> OpenStack code (to logging or to web frameworks). Since the logging code
> >> is part of Oslo and the API code can be, this seemed to provide
> >> isolation while removing most of the magic.
> >>
> >
> > I think this is exactly what we have right now inherited form Havana.
> > The _() returns a Message that is then translated on-demand by the API
> > or in a special Translation log handler.
> >
> > We just did not make Message look and feel enough like a str() and some
> > outside components (jsonifier in Glance and log Formatter all over) did
> > not know how to handle non text types correctly when non-ascii
> > characters were present.
> >
> > I think extending from unicode and removing all the implementations in
> > place such that the unicode implementation kick in for all magic methods
> > will solve the problems we saw at the end of Havana.
>
> I'm relatively new to OpenStack so I can't comment on prior OpenStack
> implementations but I'm a long standing veteran of Python i18n issues.
>
> What you're describing sounds a lot like problems that result from the
> fact Python's default encoding is ASCII as opposed to the more sensible
> UTF-8. I have a long write up on this issue from a few years ago but
> I'll cut to the chase. Python will attempt to automatically encode
> Unicode objects into ASCII during output which will fail if there are
> non-ASCII code points in the Unicode. Python does this is in two
> distinct contexts depending on whether destination of the output is a
> file or terminal. If it's a terminal it attempts to use the encoding
> associated with the TTY. Hence you can different results if you output
> to a TTY or a file handle.
>

That was related to the problem we had with logging and Message instances.


>
> The simple solution to many of the encoding exceptions that Python will
> throw is to override the default encoding and change it to UTF-8. But
> the default encoding is locked by site.py due to internal Python string
> optimizations which cache the default encoded version of the string so
> the encoding happens only once. Changing the default encoding would
> invalidate cached strings and there is no mechanism to deal with that,
> that's why the default encoding is locked. But you can change the
> default encoding using this trick if you do early enough during the
> module loading process:
>

I don't think we want to have force the encoding at startup. Setting the
locale properly through the environment and then using unicode objects also
solves the issue without any startup timing issues, and allows deployers to
choose the encoding for output.

Doug


>
> import sys
> reload(sys)
> sys.setdefaultencoding('utf-8')
>
> The reason this works is because site.py deletes the setdefaultencoding
> from the sys module, but after reloading sys it's available again. One
> can also use a tiny CPython module to set the default encoding without
> having to use the sys reload trick. The following illustrates the reload
> trick:
>
> $ python
> Python 2.7.3 (default, Aug  9 2012, 17:23:57)
> [GCC 4.7.1 20120720 (Red Hat 4.7.1-5)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import sys
> >>> sys.getdefaultencoding()
> 'ascii'
> >>> sys.setdefaultencoding('utf-8')
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> AttributeError: 'module' object has no attribute 'setdefaultencoding'
> >>> reload(sys)
> <module 'sys' (built-in)>
> >>> sys.setdefaultencoding('utf-8')
> >>> sys.getdefaultencoding()
> 'utf-8'
>
>
> Not fully undersanding the role of Python's default encoding and how
> it's application differs between terminal and non-terminal output can
> cause a lot of confusion and misunderstanding which can sometimes lead
> to false conclusions as to what is going wrong.
>
> If I get a chance I'll try to publicly post my write-up on Python i18n
> issues.


>
> --
> John
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131018/c2f0be6f/attachment.html>