Open Stack

On Tue, Jun 7, 2016 at 4:26 PM, Ben Meyer <ben.meyer at rackspace.com> wrote:

> On 06/07/2016 06:09 PM, Samuel Merritt wrote:
> > On 6/7/16 12:00 PM, Monty Taylor wrote:
> >> [snip]
> > >
> >> I'd rather see us focus energy on Python3, asyncio and its pluggable
> >> event loops. The work in:
> >>
> >> http://magic.io/blog/uvloop-blazing-fast-python-networking/
> >>
> >> is a great indication in an actual apples-to-apples comparison of what
> >> can be accomplished in python doing IO-bound activities by using modern
> >> Python techniques. I think that comparing python2+eventlet to a fresh
> >> rewrite in Go isn't 100% of the story. A TON of work has gone in to
> >> Python that we're not taking advantage of because we're still supporting
> >> Python2. So what I've love to see in the realm of comparative
> >> experimentation is to see if the existing Python we already have can be
> >> leveraged as we adopt newer and more modern things.
> >
> > Asyncio, eventlet, and other similar libraries are all very good for
> > performing asynchronous IO on sockets and pipes. However, none of them
> > help for filesystem IO. That's why Swift needs a golang object server:
> > the go runtime will keep some goroutines running even though some
> > other goroutines are performing filesystem IO, whereas filesystem IO
> > in Python blocks the entire process, asyncio or no asyncio.
>
> That can be modified. gevent has a tool
> (http://www.gevent.org/gevent.fileobject.html) that enables the File IO
> to be async  as well by putting the file into non-blocking mode. I've
> used it, and it works and scales well.
>
> Sadly, Python doesn't offer this by default; perhaps OpenStack can get
> that changed.
>
> $0.02
>
> Ben
>
>
The uvloop library is very new, but libuv itself uses a thread pool, so it
should work just fine with files once this functionality is implemented;
see https://github.com/MagicStack/uvloop/issues/1 and
https://nikhilm.github.io/uvbook/filesystem.html

I don't see any difference here with how Golang would implement such async
support, in terms of mapping goroutines against its own thread pool. With
respect to latency, there could even be better performance for certain
workloads with uvloop than Go. This is because of CPython's refcounting, vs
Go's GC - even with all the recent improvements in Go (
https://blog.golang.org/go15gc), Go still does stop the world.

A key piece here is that uvloop does not hold the GIL in these ops - a big
advantage that Python C extensions enjoy, including libraries they wrap,
and Cython explicitly enables with nogil support. So in particular we have
the following code in the the core uvloop function, which in turn is just
calling into libuv with its own core function, uv_run:

```
        # Although every UVHandle holds a reference to the loop,
        # we want to do everything to ensure that the loop will
        # never deallocate during the run -- so we do some
        # manual refs management.
        Py_INCREF(self)
        with nogil:
            err = uv.uv_run(self.uvloop, mode)
        Py_DECREF(self)
```

More here on nogil support in Cython:
http://docs.cython.org/src/userguide/external_C_code.html#declaring-a-function-as-callable-without-the-gil

- Jim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160607/cca0a03a/attachment.html>