Open Stack

On 19-08-28 06:30:36, Matthew Thode wrote:
> On 19-08-28 10:13:01, Sean Mooney wrote:
> > On Wed, 2019-08-28 at 09:47 +0100, Chris Dent wrote:
> > > Requirements people:
> > > 
> > > For the past few months we've been doing profiling of placement,
> > > modeling large clouds where many results (7000 resource providers)
> > > will be returned in response to common requests.
> > > 
> > > One of the last remaining chunks of slowness is serializing the very
> > > large dicts representing those providers (and associated allocation
> > > requests) to JSON (greater than 100k lines when pretty printed).
> > > Using the oslo jsonutils (which is a light wrapper over the stdlib
> > > 'json' module) approximately 25% of time is consumed by dumps().
> > > 
> > > Using orjson [1] instead, time consumption drops to 1%, so this is
> > > something it would be good to make real in placement (and perhaps
> > > other projects that have large result sets).
> > if the 25% to 1% numbers are real. e.g. the cpu time does not move to
> > somewhere else instead then i would be very interested to see this
> > used in oslo.versioned object and/or the sdk.
> > >  There's a WIP that
> > > demonstrates its use at [2].
> > > 
> > > I'm posting about it because JSON serialization seems like it might
> > > be an area that the requirements team has greater interest than some
> > > other areas.
> > > 
> > > Also, orjson is only Python 3, so use needs to be adapted for Python
> > > 2, as shown in the WIP [2].
> > or wait untill octber/novemebr when we officly drop python 2 support
> > that said from a redhat point of view all our future openstack release form
> > stien will be on rhel8 which is python36 only by default(python2 is not install on rhel8).
> > so we will be shipping openstack on python 3 stating with stien.
> > > 
> > > Otherwise it's fine: active, Apache2 or MIT license.
> > > 
> > > Thoughts?
> > > 
> > > [1] https://pypi.org/project/orjson/
> > > [2] https://review.opendev.org/#/c/674661/
> > > 
> > > 
> > > 
> > 
> > 
> 
> One thing we seek to avoid is duplicating requirements we already track.
> Though we have allowed C based python libs for speed.
> 
> Is it possible to try ujson as that's approved and in global-reqs
> already (and supported by anyjson, also in global-reqs).  It hasn't been
> updated since 2016 and simple PRs/issues are unresolved, though
> monasca-common and x/kiloeyes use it.  Anyjson itself seems to be in a
> similiar situation.
> 
> If we'd move to another c based json-lib I'd like to remove ujson (and
> possibly look at anyjson too, though it has slightly more usage).  There
> may be other json-libs in global reqs that meet your needs though.  I'd
> recommend checking it out first.
> 
> https://github.com/openstack/requirements/blob/master/global-requirements.txt
> 
> UJSON
> +--------------------------+----------------------------------------------------------+------+-------------------+
> | Repository               | Filename                                                 | Line | Text              |
> +--------------------------+----------------------------------------------------------+------+-------------------+
> | openstack/monasca-common | requirements.txt                                         |   11 | ujson>=1.35 # BSD |
> | x/kiloeyes               | requirements.txt                                         |   19 | ujson>=1.33       |
> +--------------------------+----------------------------------------------------------+------+-------------------+
> 
> ANYJSON
> +-----------------------------+---------------------------------------------------------------------+------+-----------------------+
> | Repository                  | Filename                                                            | Line | Text                  |
> +-----------------------------+---------------------------------------------------------------------+------+-----------------------+
> | openstack/faafo             | requirements.txt                                                    |    5 | anyjson>=0.3.3        |
> | openstack/fuel-qa           | fuelweb_test/requirements.txt                                       |    5 | anyjson>=0.3.3  # BSD |
> | openstack/fuel-web          | nailgun/requirements.txt                                            |    8 | anyjson>=0.3.3        |
> | openstack/murano-agent      | requirements.txt                                                    |    5 | anyjson>=0.3.3 # BSD  |
> | openstack/os-apply-config   | requirements.txt                                                    |    6 | anyjson>=0.3.3 # BSD  |
> | openstack/os-collect-config | requirements.txt                                                    |    6 | anyjson>=0.3.3 # BSD  |
> | openstack/os-net-config     | requirements.txt                                                    |    5 | anyjson>=0.3.3 # BSD  |
> | openstack/tacker            | requirements.txt                                                    |    9 | anyjson>=0.3.3 # BSD  |
> | starlingx/config            | sysinv/sysinv/sysinv/requirements.txt                               |    4 | anyjson>=0.3.3        |
> | starlingx/ha                | service-mgmt-client/sm-client/requirements.txt                      |    2 | anyjson>=0.3.3        |
> | starlingx/metal             | inventory/inventory/requirements.txt                                |    8 | anyjson>=0.3.3        |
> | x/apmec                     | requirements.txt                                                    |    9 | anyjson>=0.3.3 # BSD  |
> | x/daisycloud-core           | code/daisy/requirements.txt                                         |    9 | anyjson>=0.3.3        |
> | x/novajoin                  | test-requirements.txt                                               |    8 | anyjson>=0.3.3 # BSD  |
> | x/omni                      | creds_manager/test-requirements.txt                                 |    8 | anyjson>=0.3.3 # BSD  |
> +-----------------------------+---------------------------------------------------------------------+------+-----------------------+
> 

Looking through global-reqs, some more it looks like simplejson can be c
based and is in global-reqs as well.

Looking at the benchmarks orjson itself provides it's hard to see how it
can be 100x faster than even the native json.

-- 
Matthew Thode
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190828/257b0c85/attachment.sig>