Open Stack

Samuel Merritt wrote:
> On 6/7/16 12:00 PM, Monty Taylor wrote:
>> [snip]
>  >
>> I'd rather see us focus energy on Python3, asyncio and its pluggable
>> event loops. The work in:
>>
>> http://magic.io/blog/uvloop-blazing-fast-python-networking/
>>
>> is a great indication in an actual apples-to-apples comparison of what
>> can be accomplished in python doing IO-bound activities by using modern
>> Python techniques. I think that comparing python2+eventlet to a fresh
>> rewrite in Go isn't 100% of the story. A TON of work has gone in to
>> Python that we're not taking advantage of because we're still supporting
>> Python2. So what I've love to see in the realm of comparative
>> experimentation is to see if the existing Python we already have can be
>> leveraged as we adopt newer and more modern things.
>
> Asyncio, eventlet, and other similar libraries are all very good for
> performing asynchronous IO on sockets and pipes. However, none of them
> help for filesystem IO. That's why Swift needs a golang object server:
> the go runtime will keep some goroutines running even though some other
> goroutines are performing filesystem IO, whereas filesystem IO in Python
> blocks the entire process, asyncio or no asyncio.

As you probably know by now, there was a majority of TC members to deny 
the addition of Golang as another generally available tool in our 
toolbelt. The resolution (which was made to follow the guidelines the TC 
itself set as the process to follow in that specific case) was therefore 
rejected.

What made this decision pretty hard for everyone involved is that the 
Swift team built a compelling argument (see just above) on why they 
can't really solve a real user problem they have (scale and perform 
better) using the tools they are allowed to use in "OpenStack". There 
was probably also a majority of TC members agreeing that Swift needs to 
use another language to solve the specific filesystem IO problem they 
are running into. So we have been looking at various trade-off solutions 
to get the projects the tools they really need without making 
cross-project work significantly more complex, and without triggering 
the gigantic distraction of superfluous rewrites across OpenStack projects.

None of those middle-ground solutions is optimal. But I've been asked 
to list them for clarity.

One option is to develop generally-useful libraries or services outside 
of OpenStack that solve the specific issue you're running into. 
OpenStack relies on a lot of dependencies in the wider open source 
community, we don't have to develop *everything* as official "openstack 
projects" under TC governance. We even have a number of them that are 
using OpenStack infrastructure (git, gerrit, gate...). The benefit of 
this approach is that it lets you pick whatever language or framework is 
most appropriate, and forces clean contracts between the components, 
which makes replacing the external piece easier in the future. The 
drawback of this approach is that it creates local fragmentation at the 
project team level, as they have to artificially split the project into 
"OpenStack" and non-"OpenStack" bits. And it is also not optimal for 
Swift due to the way it's currently designed, you can't really slice it 
in this way.

Another option (raised by dims) is to find a way to allow usage of 
golang (or another language) in a more granular way: selectively allow 
projects which really need another tool to use it. The benefit is that 
it lets project teams make a case and use the best tool for the job, 
while limiting the cross-project impact and without opening the 
distraction floodgates of useless rewrites. The drawback is that 
depending on how it's done, it may place the TC in the role of granting 
"you're tall enough to use Go" badges, creating new endless discussions 
and more "you're special" exceptions. That said, I'm still interested in 
exploring that option, for one reason. I think that whenever a project 
team considers adding a component or a rewrite in another language, they 
are running into an interesting problem with Python, on which they 
really could use advice from the rest of the OpenStack community. I'd 
really like to see a cross-project team of Python performance experts to 
look at the problem this specific team has that makes them want to use 
another language. That seems like a great way to drive more practice 
sharing and reduce fragmentation in "OpenStack" in general. We might 
just need to put the bar pretty high so that we are not flooded by silly 
rewrite requests.

For completeness I'll also list the nuclear option: to reject the 
Technical Committee oversight altogether and decide that your project is 
better off as an autonomous project in the OpenStack ecosystem.

-- 
Thierry Carrez (ttx)