Open Stack

Hi All,

Recently, we found the server which hosts horizon dashboard had serveral
times OOM caused by horizon services. After restarting the dashboard, the
memory usage goes up very quickly if we access /project/network_topology/
path.

*How to reproduce*

Login into the dashboard and go to 'Network Topology' tab, then leave it
there (autorefresh 10s by default), now monitor the memory changes on the
host.

*Versions and Components*

Dashboard: Stable/Pike
Server: uWSGI 1.9.17-1
OS: Ubuntu 14.04 trusty
Python: 2.7.6

As the codes of memoized has little changes since Pike, if you use
Queen/Rocky release, you may also succeed to reproduce it.

*The investigation*

The root cause of the memory leak is the decorator
memorized(horizon/utils/memoized.py) which is used to cache function calls
in Horizon.

After disable it, the memory increases has been controlled.

The following is the comparison of memory change(with guppy) for each
request of /project/network_topology:

 - original (no code change) 684kb

 - do garbage collection manually 185kb

 - disable memorize cache 10kb

As we known, memoized uses weakref to cache objects. A weak reference to an
object is not enough to keep the object alive: when the only remaining
references to a referent are weak references, garbage collection is free to
destroy the referent and reuse its memory for something else.

In the memory, we could see lots of weakref stuffs, the following is a
example:




*Partition of a set of 394 objects. Total size = 37824 bytes. Index Count %
Size % Cumulative % Kind (class / dict of class)     0 197 50 18912 50
18912 50 _cffi_backend.CDataGCP     1 197 50 18912 50 37824 100
weakref.KeyedRefq*

But the rest of them are not. the following result is the memory objects
changes of per /project/network_topology access with garbage collection
manually.












*Partition of a set of 1017 objects. Total size = 183680 bytes. Index Count
% Size % Cumulative % Referrers by Kind (class / dict of class)     0 419
41 58320 32 58320 32 dict (no owner)     1 100 10 23416 13 81736 44
list     2 135 13 15184 8 96920 53 <Nothing>     3 2 0 6704 4 103624 56
urllib3.connection.VerifiedHTTPSConnection     4 2 0 6704 4 110328 60
urllib3.connectionpool.HTTPSConnectionPool     5 1 0 3352 2 113680 62
novaclient.v2.client.Client     6 2 0 2096 1 115776 63
OpenSSL.SSL.Connection     7 2 0 2096 1 117872 64 OpenSSL.SSL.Context     8
2 0 2096 1 119968 65 Queue.LifoQueue     9 12 1 2096 1 122064 66 dict of
urllib3.connectionpool.HTTPSConnectionPool*

The most of them are dicts. Followings are the dicts sorted by class, as
you can see most of them are not weakref objects:








*Partition of a set of 419 objects. Total size = 58320 bytes. Index Count %
Size % Cumulative % Class     0 362 86 50712 87 50712 87 unicode     1 27 6
3736 6 54448 93 list     2 5 1 2168 4 56616 97 dict     3 22 5 1448 2 58064
100 str     4 2 0 192 0 58256 100 weakref.KeyedRef     5 1 0 64 0 58320 100
keystoneauth1.discover.Discover*

*The issue*

So the problem is that memoized does not work like what we expect. It
allocates memory to cache objects but some of them could not be released.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20180920/8a8b15c7/attachment.html>