<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
On 2/13/12 7:00 PM, Joshua Harlow wrote:
<blockquote cite="mid:CB5EF22A.27442%25harlowja@yahoo-inc.com"
type="cite">
<title>Re: [Openstack] Nova string encoding</title>
<font face="Calibri, Verdana, Helvetica, Arial"><span
style="font-size: 11pt;">Isn’t the command line interface just
a setting on the “terminal” app you are using?<br>
</span></font></blockquote>
I'm sorry if I wasn't clear before. What's happening is: I am using
a utf8 shell (which is, I believe, normal.) Nova-manage is
receiving an argument and storing it as an 8-bit 'string'. That is
already wrong, because we've now lost track of what kind of 8-bit
string it is. Some parts of the code probably interpret it as UTF8,
but the code in the bug I'm encountering is interpreting it as
ASCII. The 'string' type in Python 2 is known to be ambiguous in
this way. Because UTF8 and ASCII overlap for certain values, this
ambiguity is seldom encountered by Americans.<br>
<br>
One solution to this is to just declare "All strings in Nova will
henceforth be treated as UTF8." That may be the current intent, but
it is not actually the case. It's also not a great policy because
it would have to be enforced 'by hand' due to Python 2's ongoing
ignorance about encodings.<br>
<br>
A more correct design which allows for future flexibility would look
like this:<br>
<br>
1) Adopt a standard for what encoding is used for all
implicitly-encoded IO. (I would propose that that standard be UTF8
rather than ASCII.)<br>
<br>
2) At all points where strings enter Python (e.g. commandline args)
immediately decode them into unicode (which can unambiguously
contain all possible 8-bit encodings.)<br>
<br>
3) At all points where 'unicodes' exit Python (being written to
stdout or a log file or a database) explicitly encode them as
appropriate (generally UTF8, again, especially if we're ever going
to read them back in.)<br>
<br>
That approach is the one I'm most familiar with, and the one
advocated for here: <a class="moz-txt-link-freetext" href="http://farmdev.com/talks/unicode/">http://farmdev.com/talks/unicode/</a>.<br>
<br>
So...back to my original question about what the policy is: Can I
assume that the answer is "There is no policy regarding string
encoding but we've been lucky so far"?<br>
<br>
-Andrew<br>
<br>
<blockquote cite="mid:CB5EF22A.27442%25harlowja@yahoo-inc.com"
type="cite"><font face="Calibri, Verdana, Helvetica, Arial"><span
style="font-size:11pt">
At least on a mac there is a
terminal->preferences->advanced which specifies which
encoding to use (mine is UTF-8).<br>
<br>
Was that tried/verified?<br>
<br>
On 2/13/12 3:52 PM, "Andrew Bogott" <<a
moz-do-not-send="true" href="abogott@wikimedia.org">abogott@wikimedia.org</a>>
wrote:<br>
<br>
</span></font>
<blockquote><font face="Calibri, Verdana, Helvetica, Arial"><span
style="font-size:11pt">On 2/13/12 5:04 PM, Naveed Massjouni
wrote:<br>
> Very recently, a change got in that converts all tables
(except 1) to<br>
> utf8 encoding, for the mysql engine. I manually tested
creating<br>
> servers with unicode names and with unicode metadata,
and it worked<br>
> fine. Make sure you are running against the latest
code. -Naveed<br>
<br>
That's a step in the right direction, but doesn't completely
address<br>
what I'm asking, unless by 'all tables' you meant 'all
tables and also<br>
all internal variables and also all REST and Commandline
interfaces.' <br>
Fixing my particular issue is straightforward, but the fact
that I'm<br>
seeing the bug in the first place suggests that there's no
standard<br>
encoding currently enforced. Which seems bad.<br>
<br>
<br>
<br>
_______________________________________________<br>
Mailing list: <a moz-do-not-send="true"
href="https://launchpad.net/%7Eopenstack">https://launchpad.net/~openstack</a><br>
Post to : <a moz-do-not-send="true"
href="openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a moz-do-not-send="true"
href="https://launchpad.net/%7Eopenstack">https://launchpad.net/~openstack</a><br>
More help : <a moz-do-not-send="true"
href="https://help.launchpad.net/ListHelp">https://help.launchpad.net/ListHelp</a><br>
<br>
</span></font></blockquote>
</blockquote>
<br>
</body>
</html>