On 19-08-28 06:30:36, Matthew Thode wrote:
On 19-08-28 10:13:01, Sean Mooney wrote:
Requirements people:
For the past few months we've been doing profiling of placement, modeling large clouds where many results (7000 resource providers) will be returned in response to common requests.
One of the last remaining chunks of slowness is serializing the very large dicts representing those providers (and associated allocation requests) to JSON (greater than 100k lines when pretty printed). Using the oslo jsonutils (which is a light wrapper over the stdlib 'json' module) approximately 25% of time is consumed by dumps().
Using orjson [1] instead, time consumption drops to 1%, so this is something it would be good to make real in placement (and perhaps other projects that have large result sets). if the 25% to 1% numbers are real. e.g. the cpu time does not move to somewhere else instead then i would be very interested to see this used in oslo.versioned object and/or the sdk. There's a WIP that demonstrates its use at [2].
I'm posting about it because JSON serialization seems like it might be an area that the requirements team has greater interest than some other areas.
Also, orjson is only Python 3, so use needs to be adapted for Python 2, as shown in the WIP [2]. or wait untill octber/novemebr when we officly drop python 2 support
On Wed, 2019-08-28 at 09:47 +0100, Chris Dent wrote: that said from a redhat point of view all our future openstack release form stien will be on rhel8 which is python36 only by default(python2 is not install on rhel8). so we will be shipping openstack on python 3 stating with stien.
Otherwise it's fine: active, Apache2 or MIT license.
Thoughts?
[1] https://pypi.org/project/orjson/ [2] https://review.opendev.org/#/c/674661/
One thing we seek to avoid is duplicating requirements we already track. Though we have allowed C based python libs for speed.
Is it possible to try ujson as that's approved and in global-reqs already (and supported by anyjson, also in global-reqs). It hasn't been updated since 2016 and simple PRs/issues are unresolved, though monasca-common and x/kiloeyes use it. Anyjson itself seems to be in a similiar situation.
If we'd move to another c based json-lib I'd like to remove ujson (and possibly look at anyjson too, though it has slightly more usage). There may be other json-libs in global reqs that meet your needs though. I'd recommend checking it out first.
https://github.com/openstack/requirements/blob/master/global-requirements.tx...
UJSON +--------------------------+----------------------------------------------------------+------+-------------------+ | Repository | Filename | Line | Text | +--------------------------+----------------------------------------------------------+------+-------------------+ | openstack/monasca-common | requirements.txt | 11 | ujson>=1.35 # BSD | | x/kiloeyes | requirements.txt | 19 | ujson>=1.33 | +--------------------------+----------------------------------------------------------+------+-------------------+
ANYJSON +-----------------------------+---------------------------------------------------------------------+------+-----------------------+ | Repository | Filename | Line | Text | +-----------------------------+---------------------------------------------------------------------+------+-----------------------+ | openstack/faafo | requirements.txt | 5 | anyjson>=0.3.3 | | openstack/fuel-qa | fuelweb_test/requirements.txt | 5 | anyjson>=0.3.3 # BSD | | openstack/fuel-web | nailgun/requirements.txt | 8 | anyjson>=0.3.3 | | openstack/murano-agent | requirements.txt | 5 | anyjson>=0.3.3 # BSD | | openstack/os-apply-config | requirements.txt | 6 | anyjson>=0.3.3 # BSD | | openstack/os-collect-config | requirements.txt | 6 | anyjson>=0.3.3 # BSD | | openstack/os-net-config | requirements.txt | 5 | anyjson>=0.3.3 # BSD | | openstack/tacker | requirements.txt | 9 | anyjson>=0.3.3 # BSD | | starlingx/config | sysinv/sysinv/sysinv/requirements.txt | 4 | anyjson>=0.3.3 | | starlingx/ha | service-mgmt-client/sm-client/requirements.txt | 2 | anyjson>=0.3.3 | | starlingx/metal | inventory/inventory/requirements.txt | 8 | anyjson>=0.3.3 | | x/apmec | requirements.txt | 9 | anyjson>=0.3.3 # BSD | | x/daisycloud-core | code/daisy/requirements.txt | 9 | anyjson>=0.3.3 | | x/novajoin | test-requirements.txt | 8 | anyjson>=0.3.3 # BSD | | x/omni | creds_manager/test-requirements.txt | 8 | anyjson>=0.3.3 # BSD | +-----------------------------+---------------------------------------------------------------------+------+-----------------------+
Looking through global-reqs, some more it looks like simplejson can be c based and is in global-reqs as well. Looking at the benchmarks orjson itself provides it's hard to see how it can be 100x faster than even the native json. -- Matthew Thode