Open Stack

On 10 May 2016 at 10:54, John Dickinson <me at not.mn> wrote:
> On 9 May 2016, at 13:16, Gregory Haynes wrote:
>>
>> This is a bit of an aside but I am sure others are wondering the same
>> thing - Is there some info (specs/etherpad/ML thread/etc) that has more
>> details on the bottleneck you're running in to? Given that the only
>> clients of your service are the public facing DNS servers I am now even
>> more surprised that you're hitting a python-inherent bottleneck.
>
> In Swift's case, the summary is that it's hard[0] to write a network
> service in Python that shuffles data between the network and a block
> device (hard drive) and effectively utilizes all of the hardware
> available. So far, we've done very well by fork()'ing child processes,
...
> Initial results from a golang reimplementation of the object server in
> Python are very positive[1]. We're not proposing to rewrite Swift
> entirely in Golang. Specifically, we're looking at improving object
> replication time in Swift. This service must discover what data is on
> a drive, talk to other servers in the cluster about what they have,
> and coordinate any data sync process that's needed.
>
> [0] Hard, not impossible. Of course, given enough time, we can do
>  anything in a Turing-complete language, right? But we're not talking
>  about possible, we're talking about efficient tools for the job at
>  hand.
...

I'm glad you're finding you can get good results in (presumably)
clean, understandable code.

Given go's historically poor perfornance with multiple cores
(https://golang.org/doc/faq#Why_GOMAXPROCS) I'm going to presume the
major advantage is in the CSP programming model - something that
Twisted does very well: and frustratingly we've had numerous
discussions from folk in the Twisted world who see the pain we have
and want to help, but as a community we've consistently stayed with
eventlet, which has a threaded programming model - and threaded models
are poorly suited for the case here.

All of which to say, I disagree quite strongly that its hard to drive
a server's hardware at full capacity using Python. Using Python like
*we have been* - yes, certainly.

Thats not a reason to avoid golang - but if minimising the number of
languages in play is a thing we want to do, we should make sure we're
actually using the best facilities of our existing languages first!

On the Designate side, PyPy got a 40% performance improvement over
CPython for a benchmark of Twisted's DNS component 6 years ago - and
Twisted and PyPy have both moved forward a great deal since then....
http://morepypy.blogspot.co.nz/2010/03/hello.html

-Rob

-- 
Robert Collins <rbtcollins at hpe.com>
Distinguished Technologist
HP Converged Cloud