Open Stack

Thanks John for the intro.

I believe cryptography is supported by pypy today.   Just did a "pip install cryptography" using an older version of pypy, pypy2-v5.6.0, with no error.  This package has been downloaded > 13 millions of times based on pypy package tracking site: http://packages.pypy.org/##cryptography.  
I'd be interested in knowing the exact errors where it's failing, and help out.

As John said, we've observed 2x boost in throughput, and 78% reduction in latency from the proxy node in Swift lab setups.
We've also seen perf gain from Cinder, Keystone, Nova, Glance, Neutron, with most significant gains seen (22x) from Oslo.i18n from our lab setup running benchmarks such as Rally.

Switching from CPython to PyPy for OpenStack is not hard.  However, I found it really challenging to identify real world (customers) OpenStack examples where performance is hindered mainly on Python codes/or interpreter.

Thanks,

Peter


-----Original Message-----
From: John Dickinson [mailto:me at not.mn] 
Sent: Wednesday, May 31, 2017 9:22 AM
To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
Cc: Wang, Peter Xihong <peter.xihong.wang at intel.com>
Subject: Re: [openstack-dev] [requirements] Do we care about pypy for clients (broken by cryptography)



On 31 May 2017, at 5:34, Monty Taylor wrote:

> On 05/31/2017 06:39 AM, Sean McGinnis wrote:
>> On Wed, May 31, 2017 at 06:37:02AM -0500, Sean McGinnis wrote:
>>> We had a discussion a few months back around what to do for 
>>> cryptography since pycrypto is basically dead [1]. After some 
>>> discussion, at least on the Cinder project, we decided the best way 
>>> forward was to use the cryptography package instead, and work has 
>>> been done to completely remove pycrypto usage.
>>>
>>> It all seemed like a good plan at the time.
>>>
>>> I now notice that for the python-cinderclient jobs, there is a pypy 
>>> job
>>> (non-voting!) that is failing because the cryptography package is 
>>> not supported with pypy.
>>>
>>> So this leaves us with two options I guess. Change the cryto library 
>>> again, or drop support for pypy.
>>>
>>> I am not aware of anyone using pypy, and there are other valid 
>>> working alternatives. I would much rather just drop support for it 
>>> than redo our crypto functions again.
>>>
>>> Thoughts? I'm sure the Grand Champion of the Clients (Monty) 
>>> probably has some input?
>
> There was work a few years ago to get pypy support going - but it never really seemed to catch on. The chance that we're going to start a new push and be successful at this point seems low at best.
>
> I'd argue that pypy is already not supported, so dropping the non-voting job doesn't seem like losing very much to me. Reworking cryptography libs again, otoh, seems like a lot of work.
>
> Monty

On the other hand, I've been working with Intel on getting PyPy support in Swift (it works, just need to reenable that gate job), and I know they were working on support in other projects. Summary is that for Swift, we got around 2x improvement (lower latency + higher throughput) by simply replacing CPython with PyPy. I believe similar gains in other projects were expected.

I've cc'd Peter from Intel who's been working on PyPy quite a bit. I know he'll be very interested in this discussion and will have valuable input.

--john




>
> ______________________________________________________________________
> ____ OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev