Open Stack

On 15:54 May 09, John Dickinson wrote:
> On 9 May 2016, at 13:16, Gregory Haynes wrote:
> >
> > This is a bit of an aside but I am sure others are wondering the same
> > thing - Is there some info (specs/etherpad/ML thread/etc) that has more
> > details on the bottleneck you're running in to? Given that the only
> > clients of your service are the public facing DNS servers I am now even
> > more surprised that you're hitting a python-inherent bottleneck.
> 
> In Swift's case, the summary is that it's hard[0] to write a network
> service in Python that shuffles data between the network and a block
> device (hard drive) and effectively utilizes all of the hardware
> available. So far, we've done very well by fork()'ing child processes,
> using cooperative concurrency via eventlet, and basic "write more
> efficient code" optimizations. However, when it comes down to it,
> managing all of the async operations across many cores and many drives
> is really hard, and there just isn't a good, efficient interface for
> that in Python.

If I'm understanding correctly you're findings are:

1) Performance great from benchmarks.
2) Interface great for dealing with network and block devices

For item one, I'm wondering if asyncio was explored at all? I get more and more
curious if this is going to be a thing in the past [1] as I read some
improvements in this area (still immature as noted).

For item two, can you speak more about the interface improvements with working
with all the hardware available. Cinder for example, we deal with block devices
a bit and having to sometimes just dd data from here to there.

[1] - http://magic.io/blog/uvloop-blazing-fast-python-networking/

-- 
Mike Perez