Open Stack

You're absolutely right. If we can get as good (or even close) to the same performance eg with PyPy, then we should absolutely do that! I've had many public and private conversations over the last year or so that have that same basic message as I've been looking at the ongoing Golang work in the Swift community.

In fact, for the last several months I have been working pretty closely with some Intel engineers who are dedicated to Python (and specifically PyPy) performance improvements. We've been running Swift under PyPy and there's been other more preliminary tests with other OpenStack projects. I gave a joint talk in Austin about this work: https://www.youtube.com/watch?v=L5sCD7zEENg

Based on that testing, I have a few conclusions:

1) (assuming production-ready stability of PyPy and OpenStack running under it) Everyone should be running OpenStack projects under PyPy instead of CPython. Put simply, it's just plain faster. There are some issues to work out, though, specifically relating to PyPy's GC. Right now, we're looking at a few patches that do a better job of socket handling in Swift so that it runs better under PyPy. Once these patches land, I really do hope that more people will be using PyPy in production.

2) PyPy only helps when you've got a CPU-constrained environment.

3) The Golang targets in Swift are related to effective thread management, syscalls, and IO (as Sam described in a few earlier posts in this thread), and these are the issues we're facing on the persistent storage layer in Swift.

The conclusion is that there will still be Python code in Swift for a long time to come, and there is serious effort underway to make sure we can run that in production on PyPy and get some great performance improvements there. However, PyPy is not helping us when it comes to the persistent storage layer in Swift. For that, we need scalable, lightweight, and fast syscall management and thread coordination (again, see Sam's earlier messages). Perhaps we could write some Python C extension to get something like that, but (1) the best we'd hope for would only be approaching what we'd get with Golang's runtime out-of-the box and (2) we'd still be having this exact same conversation except about C.

So I expect that a year from now (or sooner) Swift deployers will be running both Python code with PyPy and compiled Golang code. And they'll have a storage system that's a lot faster than is possible today.


--John





On 13 May 2016, at 2:04, Alexey Stupnikov wrote:

> + Agree. It is strange to use another language to address performance issues if you haven't tried to solve those issues using original language's options.
>
> On 05/13/2016 11:53 AM, Fausto Marzi wrote:
>> ++ Brilliant.
>>
>> On Fri, May 13, 2016 at 10:14 AM, Dmitry Tantsur <dtantsur at redhat.com <mailto:dtantsur at redhat.com>> wrote:
>>
>>
>>     This is pretty subjective, I would say. I personally don't feel Go
>>     (especially its approach to error handling) any natural (at least
>>     no more than Rust or Scala, for example). If familiarity for
>>     Python developers is an argument here, mastering Cython or making
>>     OpenStack run on PyPy must be much easier for a random Python
>>     developer out there to seriously bump the performance. And it
>>     would not require introducing a completely new language to the
>>     picture.
>>
>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> -- 
> BR, Alexey Stupnikov.
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160513/3e16e205/attachment.pgp>