Open Stack

On Wed, Feb 05, 2020 at 04:21:43PM -0600, Matthew Thode wrote:
> On 20-02-05 16:38:26, Matthew Treinish wrote:
> > On Wed, Feb 05, 2020 at 10:46:30AM -0600, Matthew Thode wrote:
> > > On 20-02-05 10:17:18, Witek Bedyk wrote:
> > > > Hi,
> > > > 
> > > > we're using ujson in Monasca for serialization all over the place. While
> > > > changing it to any other alternative is probably a drop-in replacement, we
> > > > have in the past chosen to use ujson because of better performance. It is of
> > > > great importance, in particular in persister. Current alternatives include
> > > > orjson [1] and rapidjson [2]. We're going to measure which of them works
> > > > best for our use case and how much faster they are compared to standard
> > > > library module.
> > > > 
> > > > Assuming there is a significant performance benefit, is there any preference
> > > > from requirements team which one to include in global requirements? I
> > > > haven't seen any distro packages for any of them.
> > > > 
> > > > [1] https://pypi.org/project/orjson/
> > > > [2] https://pypi.org/project/python-rapidjson/
> > > > 
> > > > Best greetings
> > > > Witek
> > > > 
> > > > 
> > > > On 1/31/20 9:34 AM, Radosław Piliszek wrote:
> > > > > This is a spinoff discussion of [1] to attract more people.
> > > > > 
> > > > > As the subject goes, the situation of ujson is bad. Still, monasca and
> > > > > gnocchi (both server and client) seem to be using it which may break
> > > > > depending on compiler.
> > > > > The original issue is that the released version of ujson is in
> > > > > non-spec-conforming C which may break randomly based on used compiler
> > > > > and linker.
> > > > > There has been no release of ujson for more than 4 years.
> > > > > 
> > > > > Based on general project activity, Monasca is probably able to fix it
> > > > > but Gnocchi not so surely...
> > > > > 
> > > > > [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-January/thread.html
> > > > > 
> > > > > -yoctozepto
> > > > > 
> > > > 
> > > 
> > > orjson supports CPython 3.6, 3.7, 3.8, and 3.9. It distributes wheels
> > > for Linux, macOS, and Windows. The manylinux1 wheel differs from PEP 513
> > > in requiring glibc 2.18, released 2013, or later. orjson does not
> > > support PyPy.
> > > 
> > > Given the above (I think we still need to support py35 at least) I'm not
> > > sure we can use it.  Though it is my preferred other than that...
> > > (faster than ujson, more updates (last release yesterday), etc)
> > 
> > It's also probably worth looking at the thread on this from August [1]
> > discussing potentially using orjson. While they've added sdists since that
> > original discussion (because of the pyo3-pack support being added) building
> > it locally requires having rust nightly installed. This means for anyone on
> > a non-x86_64 platform (including i686) will need to have rust nightly
> > installed to pip install a package. Not that it's a super big burden, rustup
> > makes it pretty easy to do, but it's a pretty uncommon thing for most people.
> > But, I think that combined with no py35 support probably makes it a difficult
> > thing to add to g-r. 
> > 
> > [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008849.html
> > 
> > -Matthew Treinish
> 
> Forgot about that, does it still need nightly though?  I'd hope that it
> doesn't but wouldn't be surprised if it does.  Arch support is important
> though, some jobs execute on ppc64 and arm64 as well iirc.
> 

It does, it's because of PyO3 [1] which is the library most projects use for
a python<->rust interface. PyO3 relies on some rust language features which
are not part of stable yet. That being said I've got a couple rust/python
projects[2][3] and have never had an issue with nightly rust, it's surprisingly
stable and useable, and rustup makes installing it and keeping up to date
simple. But despite that, for a project the size of OpenStack I think it's
probably a bit much to ask for anyone not on x86_64 to need to have rust
nightly installed just to install things via pip.

-Matthew Treinish

[1] https://github.com/PyO3/pyo3
[2] https://github.com/mtreinish/retworkx
[3] https://github.com/mtreinish/pyrqasm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200205/8120c92d/attachment-0001.sig>