[all][requirements][monasca][gnocchi] ujson, not maintained for over 4 years, has compiler issues
This is a spinoff discussion of [1] to attract more people. As the subject goes, the situation of ujson is bad. Still, monasca and gnocchi (both server and client) seem to be using it which may break depending on compiler. The original issue is that the released version of ujson is in non-spec-conforming C which may break randomly based on used compiler and linker. There has been no release of ujson for more than 4 years. Based on general project activity, Monasca is probably able to fix it but Gnocchi not so surely... [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-January/thread.h... -yoctozepto
I looked into that some days ago and the ujson usage is pretty minimal in Gnocchi. Digging up the commits that changed to ujson (wasn't that long ago I think, for performance reasons) and revert those should be too much work. Best regards On 1/31/20 9:38 AM, Radosław Piliszek wrote:
This is a spinoff discussion of [1] to attract more people.
As the subject goes, the situation of ujson is bad. Still, monasca and gnocchi (both server and client) seem to be using it which may break depending on compiler. The original issue is that the released version of ujson is in non-spec-conforming C which may break randomly based on used compiler and linker. There has been no release of ujson for more than 4 years.
Based on general project activity, Monasca is probably able to fix it but Gnocchi not so surely...
[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-January/thread.h...
-yoctozepto
Hi, we're using ujson in Monasca for serialization all over the place. While changing it to any other alternative is probably a drop-in replacement, we have in the past chosen to use ujson because of better performance. It is of great importance, in particular in persister. Current alternatives include orjson [1] and rapidjson [2]. We're going to measure which of them works best for our use case and how much faster they are compared to standard library module. Assuming there is a significant performance benefit, is there any preference from requirements team which one to include in global requirements? I haven't seen any distro packages for any of them. [1] https://pypi.org/project/orjson/ [2] https://pypi.org/project/python-rapidjson/ Best greetings Witek On 1/31/20 9:34 AM, Radosław Piliszek wrote:
This is a spinoff discussion of [1] to attract more people.
As the subject goes, the situation of ujson is bad. Still, monasca and gnocchi (both server and client) seem to be using it which may break depending on compiler. The original issue is that the released version of ujson is in non-spec-conforming C which may break randomly based on used compiler and linker. There has been no release of ujson for more than 4 years.
Based on general project activity, Monasca is probably able to fix it but Gnocchi not so surely...
[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-January/thread.h...
-yoctozepto
On 20-02-05 10:17:18, Witek Bedyk wrote:
Hi,
we're using ujson in Monasca for serialization all over the place. While changing it to any other alternative is probably a drop-in replacement, we have in the past chosen to use ujson because of better performance. It is of great importance, in particular in persister. Current alternatives include orjson [1] and rapidjson [2]. We're going to measure which of them works best for our use case and how much faster they are compared to standard library module.
Assuming there is a significant performance benefit, is there any preference from requirements team which one to include in global requirements? I haven't seen any distro packages for any of them.
[1] https://pypi.org/project/orjson/ [2] https://pypi.org/project/python-rapidjson/
Best greetings Witek
On 1/31/20 9:34 AM, Radosław Piliszek wrote:
This is a spinoff discussion of [1] to attract more people.
As the subject goes, the situation of ujson is bad. Still, monasca and gnocchi (both server and client) seem to be using it which may break depending on compiler. The original issue is that the released version of ujson is in non-spec-conforming C which may break randomly based on used compiler and linker. There has been no release of ujson for more than 4 years.
Based on general project activity, Monasca is probably able to fix it but Gnocchi not so surely...
[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-January/thread.h...
-yoctozepto
orjson supports CPython 3.6, 3.7, 3.8, and 3.9. It distributes wheels for Linux, macOS, and Windows. The manylinux1 wheel differs from PEP 513 in requiring glibc 2.18, released 2013, or later. orjson does not support PyPy. Given the above (I think we still need to support py35 at least) I'm not sure we can use it. Though it is my preferred other than that... (faster than ujson, more updates (last release yesterday), etc) -- Matthew Thode
orjson supports CPython 3.6, 3.7, 3.8, and 3.9. It distributes wheels for Linux, macOS, and Windows. The manylinux1 wheel differs from PEP 513 in requiring glibc 2.18, released 2013, or later. orjson does not support PyPy.
Given the above (I think we still need to support py35 at least) I'm not sure we can use it. Though it is my preferred other than that... (faster than ujson, more updates (last release yesterday), etc)
Thanks Matthew, what's the status of Python 3.5 support? We've been dropping unit tests for 3.5 in Train [1]. [1] https://governance.openstack.org/tc/goals/selected/train/python3-updates.htm... Witek
On 20-02-05 18:09:05, Witek Bedyk wrote:
orjson supports CPython 3.6, 3.7, 3.8, and 3.9. It distributes wheels for Linux, macOS, and Windows. The manylinux1 wheel differs from PEP 513 in requiring glibc 2.18, released 2013, or later. orjson does not support PyPy.
Given the above (I think we still need to support py35 at least) I'm not sure we can use it. Though it is my preferred other than that... (faster than ujson, more updates (last release yesterday), etc)
Thanks Matthew, what's the status of Python 3.5 support? We've been dropping unit tests for 3.5 in Train [1].
[1] https://governance.openstack.org/tc/goals/selected/train/python3-updates.htm...
Witek
Looks like you are right, for some reason I thought we still supported it. looks good to me then -- Matthew Thode
On Wed, Feb 05, 2020 at 10:46:30AM -0600, Matthew Thode wrote:
On 20-02-05 10:17:18, Witek Bedyk wrote:
Hi,
we're using ujson in Monasca for serialization all over the place. While changing it to any other alternative is probably a drop-in replacement, we have in the past chosen to use ujson because of better performance. It is of great importance, in particular in persister. Current alternatives include orjson [1] and rapidjson [2]. We're going to measure which of them works best for our use case and how much faster they are compared to standard library module.
Assuming there is a significant performance benefit, is there any preference from requirements team which one to include in global requirements? I haven't seen any distro packages for any of them.
[1] https://pypi.org/project/orjson/ [2] https://pypi.org/project/python-rapidjson/
Best greetings Witek
On 1/31/20 9:34 AM, Radosław Piliszek wrote:
This is a spinoff discussion of [1] to attract more people.
As the subject goes, the situation of ujson is bad. Still, monasca and gnocchi (both server and client) seem to be using it which may break depending on compiler. The original issue is that the released version of ujson is in non-spec-conforming C which may break randomly based on used compiler and linker. There has been no release of ujson for more than 4 years.
Based on general project activity, Monasca is probably able to fix it but Gnocchi not so surely...
[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-January/thread.h...
-yoctozepto
orjson supports CPython 3.6, 3.7, 3.8, and 3.9. It distributes wheels for Linux, macOS, and Windows. The manylinux1 wheel differs from PEP 513 in requiring glibc 2.18, released 2013, or later. orjson does not support PyPy.
Given the above (I think we still need to support py35 at least) I'm not sure we can use it. Though it is my preferred other than that... (faster than ujson, more updates (last release yesterday), etc)
It's also probably worth looking at the thread on this from August [1] discussing potentially using orjson. While they've added sdists since that original discussion (because of the pyo3-pack support being added) building it locally requires having rust nightly installed. This means for anyone on a non-x86_64 platform (including i686) will need to have rust nightly installed to pip install a package. Not that it's a super big burden, rustup makes it pretty easy to do, but it's a pretty uncommon thing for most people. But, I think that combined with no py35 support probably makes it a difficult thing to add to g-r. [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008849.ht... -Matthew Treinish
On 20-02-05 16:38:26, Matthew Treinish wrote:
On Wed, Feb 05, 2020 at 10:46:30AM -0600, Matthew Thode wrote:
On 20-02-05 10:17:18, Witek Bedyk wrote:
Hi,
we're using ujson in Monasca for serialization all over the place. While changing it to any other alternative is probably a drop-in replacement, we have in the past chosen to use ujson because of better performance. It is of great importance, in particular in persister. Current alternatives include orjson [1] and rapidjson [2]. We're going to measure which of them works best for our use case and how much faster they are compared to standard library module.
Assuming there is a significant performance benefit, is there any preference from requirements team which one to include in global requirements? I haven't seen any distro packages for any of them.
[1] https://pypi.org/project/orjson/ [2] https://pypi.org/project/python-rapidjson/
Best greetings Witek
On 1/31/20 9:34 AM, Radosław Piliszek wrote:
This is a spinoff discussion of [1] to attract more people.
As the subject goes, the situation of ujson is bad. Still, monasca and gnocchi (both server and client) seem to be using it which may break depending on compiler. The original issue is that the released version of ujson is in non-spec-conforming C which may break randomly based on used compiler and linker. There has been no release of ujson for more than 4 years.
Based on general project activity, Monasca is probably able to fix it but Gnocchi not so surely...
[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-January/thread.h...
-yoctozepto
orjson supports CPython 3.6, 3.7, 3.8, and 3.9. It distributes wheels for Linux, macOS, and Windows. The manylinux1 wheel differs from PEP 513 in requiring glibc 2.18, released 2013, or later. orjson does not support PyPy.
Given the above (I think we still need to support py35 at least) I'm not sure we can use it. Though it is my preferred other than that... (faster than ujson, more updates (last release yesterday), etc)
It's also probably worth looking at the thread on this from August [1] discussing potentially using orjson. While they've added sdists since that original discussion (because of the pyo3-pack support being added) building it locally requires having rust nightly installed. This means for anyone on a non-x86_64 platform (including i686) will need to have rust nightly installed to pip install a package. Not that it's a super big burden, rustup makes it pretty easy to do, but it's a pretty uncommon thing for most people. But, I think that combined with no py35 support probably makes it a difficult thing to add to g-r.
[1] http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008849.ht...
-Matthew Treinish
Forgot about that, does it still need nightly though? I'd hope that it doesn't but wouldn't be surprised if it does. Arch support is important though, some jobs execute on ppc64 and arm64 as well iirc. -- Matthew Thode
On Wed, Feb 05, 2020 at 04:21:43PM -0600, Matthew Thode wrote:
On 20-02-05 16:38:26, Matthew Treinish wrote:
On Wed, Feb 05, 2020 at 10:46:30AM -0600, Matthew Thode wrote:
On 20-02-05 10:17:18, Witek Bedyk wrote:
Hi,
we're using ujson in Monasca for serialization all over the place. While changing it to any other alternative is probably a drop-in replacement, we have in the past chosen to use ujson because of better performance. It is of great importance, in particular in persister. Current alternatives include orjson [1] and rapidjson [2]. We're going to measure which of them works best for our use case and how much faster they are compared to standard library module.
Assuming there is a significant performance benefit, is there any preference from requirements team which one to include in global requirements? I haven't seen any distro packages for any of them.
[1] https://pypi.org/project/orjson/ [2] https://pypi.org/project/python-rapidjson/
Best greetings Witek
On 1/31/20 9:34 AM, Radosław Piliszek wrote:
This is a spinoff discussion of [1] to attract more people.
As the subject goes, the situation of ujson is bad. Still, monasca and gnocchi (both server and client) seem to be using it which may break depending on compiler. The original issue is that the released version of ujson is in non-spec-conforming C which may break randomly based on used compiler and linker. There has been no release of ujson for more than 4 years.
Based on general project activity, Monasca is probably able to fix it but Gnocchi not so surely...
[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-January/thread.h...
-yoctozepto
orjson supports CPython 3.6, 3.7, 3.8, and 3.9. It distributes wheels for Linux, macOS, and Windows. The manylinux1 wheel differs from PEP 513 in requiring glibc 2.18, released 2013, or later. orjson does not support PyPy.
Given the above (I think we still need to support py35 at least) I'm not sure we can use it. Though it is my preferred other than that... (faster than ujson, more updates (last release yesterday), etc)
It's also probably worth looking at the thread on this from August [1] discussing potentially using orjson. While they've added sdists since that original discussion (because of the pyo3-pack support being added) building it locally requires having rust nightly installed. This means for anyone on a non-x86_64 platform (including i686) will need to have rust nightly installed to pip install a package. Not that it's a super big burden, rustup makes it pretty easy to do, but it's a pretty uncommon thing for most people. But, I think that combined with no py35 support probably makes it a difficult thing to add to g-r.
[1] http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008849.ht...
-Matthew Treinish
Forgot about that, does it still need nightly though? I'd hope that it doesn't but wouldn't be surprised if it does. Arch support is important though, some jobs execute on ppc64 and arm64 as well iirc.
It does, it's because of PyO3 [1] which is the library most projects use for a python<->rust interface. PyO3 relies on some rust language features which are not part of stable yet. That being said I've got a couple rust/python projects[2][3] and have never had an issue with nightly rust, it's surprisingly stable and useable, and rustup makes installing it and keeping up to date simple. But despite that, for a project the size of OpenStack I think it's probably a bit much to ask for anyone not on x86_64 to need to have rust nightly installed just to install things via pip. -Matthew Treinish [1] https://github.com/PyO3/pyo3 [2] https://github.com/mtreinish/retworkx [3] https://github.com/mtreinish/pyrqasm
Alrighty, folks. The summary is: orjson is bad due to requirement of PyO3 to run rust nightly on non-x86_64. Then what about the other contestant, rapidjson? [1] It's a wrapper over a popular C++ json library. Both wrapper and lib look alive enough to be considered serious. [1] https://pypi.org/project/python-rapidjson/ -yoctozepto
On 20-02-06 08:35:03, Radosław Piliszek wrote:
Alrighty, folks.
The summary is: orjson is bad due to requirement of PyO3 to run rust nightly on non-x86_64.
Then what about the other contestant, rapidjson? [1] It's a wrapper over a popular C++ json library. Both wrapper and lib look alive enough to be considered serious.
[1] https://pypi.org/project/python-rapidjson/
-yoctozepto
Yep, rapidjson looks fine to me -- Matthew Thode
participants (5)
-
Matthew Thode
-
Matthew Treinish
-
Radosław Piliszek
-
Tobias Urdin
-
Witek Bedyk