Open Stack

On 19-08-28 10:13:01, Sean Mooney wrote:
> On Wed, 2019-08-28 at 09:47 +0100, Chris Dent wrote:
> > Requirements people:
> > 
> > For the past few months we've been doing profiling of placement,
> > modeling large clouds where many results (7000 resource providers)
> > will be returned in response to common requests.
> > 
> > One of the last remaining chunks of slowness is serializing the very
> > large dicts representing those providers (and associated allocation
> > requests) to JSON (greater than 100k lines when pretty printed).
> > Using the oslo jsonutils (which is a light wrapper over the stdlib
> > 'json' module) approximately 25% of time is consumed by dumps().
> > 
> > Using orjson [1] instead, time consumption drops to 1%, so this is
> > something it would be good to make real in placement (and perhaps
> > other projects that have large result sets).
> if the 25% to 1% numbers are real. e.g. the cpu time does not move to
> somewhere else instead then i would be very interested to see this
> used in oslo.versioned object and/or the sdk.
> >  There's a WIP that
> > demonstrates its use at [2].
> > 
> > I'm posting about it because JSON serialization seems like it might
> > be an area that the requirements team has greater interest than some
> > other areas.
> > 
> > Also, orjson is only Python 3, so use needs to be adapted for Python
> > 2, as shown in the WIP [2].
> or wait untill octber/novemebr when we officly drop python 2 support
> that said from a redhat point of view all our future openstack release form
> stien will be on rhel8 which is python36 only by default(python2 is not install on rhel8).
> so we will be shipping openstack on python 3 stating with stien.
> > 
> > Otherwise it's fine: active, Apache2 or MIT license.
> > 
> > Thoughts?
> > 
> > [1] https://pypi.org/project/orjson/
> > [2] https://review.opendev.org/#/c/674661/
> > 
> > 
> > 
> 
> 

One thing we seek to avoid is duplicating requirements we already track.
Though we have allowed C based python libs for speed.

Is it possible to try ujson as that's approved and in global-reqs
already (and supported by anyjson, also in global-reqs).  It hasn't been
updated since 2016 and simple PRs/issues are unresolved, though
monasca-common and x/kiloeyes use it.  Anyjson itself seems to be in a
similiar situation.

If we'd move to another c based json-lib I'd like to remove ujson (and
possibly look at anyjson too, though it has slightly more usage).  There
may be other json-libs in global reqs that meet your needs though.  I'd
recommend checking it out first.

https://github.com/openstack/requirements/blob/master/global-requirements.txt

UJSON
+--------------------------+----------------------------------------------------------+------+-------------------+
| Repository               | Filename                                                 | Line | Text              |
+--------------------------+----------------------------------------------------------+------+-------------------+
| openstack/monasca-common | requirements.txt                                         |   11 | ujson>=1.35 # BSD |
| x/kiloeyes               | requirements.txt                                         |   19 | ujson>=1.33       |
+--------------------------+----------------------------------------------------------+------+-------------------+

ANYJSON
+-----------------------------+---------------------------------------------------------------------+------+-----------------------+
| Repository                  | Filename                                                            | Line | Text                  |
+-----------------------------+---------------------------------------------------------------------+------+-----------------------+
| openstack/faafo             | requirements.txt                                                    |    5 | anyjson>=0.3.3        |
| openstack/fuel-qa           | fuelweb_test/requirements.txt                                       |    5 | anyjson>=0.3.3  # BSD |
| openstack/fuel-web          | nailgun/requirements.txt                                            |    8 | anyjson>=0.3.3        |
| openstack/murano-agent      | requirements.txt                                                    |    5 | anyjson>=0.3.3 # BSD  |
| openstack/os-apply-config   | requirements.txt                                                    |    6 | anyjson>=0.3.3 # BSD  |
| openstack/os-collect-config | requirements.txt                                                    |    6 | anyjson>=0.3.3 # BSD  |
| openstack/os-net-config     | requirements.txt                                                    |    5 | anyjson>=0.3.3 # BSD  |
| openstack/tacker            | requirements.txt                                                    |    9 | anyjson>=0.3.3 # BSD  |
| starlingx/config            | sysinv/sysinv/sysinv/requirements.txt                               |    4 | anyjson>=0.3.3        |
| starlingx/ha                | service-mgmt-client/sm-client/requirements.txt                      |    2 | anyjson>=0.3.3        |
| starlingx/metal             | inventory/inventory/requirements.txt                                |    8 | anyjson>=0.3.3        |
| x/apmec                     | requirements.txt                                                    |    9 | anyjson>=0.3.3 # BSD  |
| x/daisycloud-core           | code/daisy/requirements.txt                                         |    9 | anyjson>=0.3.3        |
| x/novajoin                  | test-requirements.txt                                               |    8 | anyjson>=0.3.3 # BSD  |
| x/omni                      | creds_manager/test-requirements.txt                                 |    8 | anyjson>=0.3.3 # BSD  |
+-----------------------------+---------------------------------------------------------------------+------+-----------------------+



-- 
Matthew Thode
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190828/7644cf19/attachment.sig>