Open Stack

On Mon, May 9, 2016, at 03:54 PM, John Dickinson wrote:
> On 9 May 2016, at 13:16, Gregory Haynes wrote:
> >
> > This is a bit of an aside but I am sure others are wondering the same
> > thing - Is there some info (specs/etherpad/ML thread/etc) that has more
> > details on the bottleneck you're running in to? Given that the only
> > clients of your service are the public facing DNS servers I am now even
> > more surprised that you're hitting a python-inherent bottleneck.
> 
> In Swift's case, the summary is that it's hard[0] to write a network
> service in Python that shuffles data between the network and a block
> device (hard drive) and effectively utilizes all of the hardware
> available. So far, we've done very well by fork()'ing child processes,
> using cooperative concurrency via eventlet, and basic "write more
> efficient code" optimizations. However, when it comes down to it,
> managing all of the async operations across many cores and many drives
> is really hard, and there just isn't a good, efficient interface for
> that in Python.

This is a pretty big difference from hitting an unsolvable performance
issue in the language and instead is a case of language preference -
which is fine. I don't really want to fall in to the language-comparison
trap, but I think more detailed reasoning for why it is preferable over
python in specific use cases we have hit is good info to include /
discuss in the document you're drafting :). Essentially its a matter of
weighing the costs (which lots of people have hit on so I won't) with
the potential benefits and so unless the benefits are made very clear
(especially if those benefits are technical) its pretty hard to evaluate
IMO.

There seemed to be an assumption in some of the designate rewrite posts
that there is some language-inherent performance issue causing a
bottleneck. If this does actually exist then that is a good reason for
rewriting in another language and is something that would be very useful
to clearly document as a case where we support this type of thing. I am
highly suspicious that this is the case though, but I am trying hard to
keep an open mind...

> 
> Initial results from a golang reimplementation of the object server in
> Python are very positive[1]. We're not proposing to rewrite Swift
> entirely in Golang. Specifically, we're looking at improving object
> replication time in Swift. This service must discover what data is on
> a drive, talk to other servers in the cluster about what they have,
> and coordinate any data sync process that's needed.
> 
> [0] Hard, not impossible. Of course, given enough time, we can do
>  anything in a Turing-complete language, right? But we're not talking
>  about possible, we're talking about efficient tools for the job at
>  hand.

Sorry to be a pedant but that is just plain false - there are plenty of
intractable problems.

> 
> [1] http://d.not.mn/python_vs_golang_gets.png and
>  http://d.not.mn/python_vs_golang_puts.png
> 
> 
> --John