Open Stack

I was looking into this with Anita Kuno today.

Keystone unit tests would fail when she was running on her vm set up for
testing. We found that the system was running out of memory. When I run it
myself, watching memory usage with top, the nosetester process uses over
800MB. Anita's vm has 1GB total, and she sent me a screenshot showing that
the oom killer got it. So it's not that the Keystone unit tests don't work
in a venv generally -- you need enough of RAM.

Memory use growing without bound indicates a memory leak, so I started
looking into where references were piling up. I was hoping it would be an
easy fix where the Keystone test code could be changed to clear a list or
something. I haven't looked at python memory leaks before so I might be
wrong on all this, but here's my analysis.

The summary is the references are piling up in the nosetests runner as it's
storing the results of all the tests, so I don't think there's anything we
can do in Keystone code to fix it.

I made use of heapy https://pypi.python.org/pypi/guppy/0.1.10 to find the
heap usage.

After running 872 tests,

> /usr/lib/python2.7/dist-packages/nose/suite.py(223)run()
-> test(orig)
(Pdb) p result
<nose.result.TextTestResult run=869 errors=3 failures=0>

hpy.heap() shows the biggest user of the heap is unnamed dicts.

(Pdb) h1.heap()
Partition of a set of 487786 objects. Total size = 88365600 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  76542  16 27581904  31  27581904  31 dict (no owner)
     1 128620  26 11649056  13  39230960  44 str
     2  72387  15  6771416   8  46002376  52 tuple
     3   3043   1  3189064   4  49191440  56 dict of 0x2b96700
     4  21977   5  2615816   3  51807256  59 list
     5  20374   4  2444880   3  54252136  61 function
     6    739   0  2435656   3  56687792  64 dict of
sqlalchemy.dialects.sqlite.base.SQLiteCompiler
     7   2579   1  2324312   3  59012104  67 type
     8  17378   4  2224384   3  61236488  69 types.CodeType
     9    854   0  2123792   2  63360280  72 dict of module
<899 more rows. Type e.g. '_.more' to view.>

By running for a while and watching the list change, one sees the "dict (no
owner)" line start out at the bottom of the list and climb up to the top as
more tests run.

Digging into the dict data a bit more, one can see what's holding the refs
to dicts,

(Pdb)
h1.heap([0].get_rp(depth=1)


Reference Pattern by <[dict of]
class>.

 0: _ --- [-] 76503 dict (no owner): 0x1a4f150*1673,
0x1a52380*15...

 1: a      [+] 739 dict of sqlalchemy.dialects.sqlite.base.SQLiteCompiler:
0x...
 2: b ---- [+] 514 dict of routes.route.Route: 0x5db1b50...,
0x68f02d0

 3: c      [+] 905 dict of nose.case.Test: 0x2a78490, 0x2ad2c90,
0x2ad2e50...
 4: d ---- [+] 873 dict (no owner): 0x1aaff60*10, 0x1bf9c90*2,
0x1f7ef10*34...

 5: e      [S] 157 dict of module: _warnings, errno..., os, sys,
zipimport
 6: f ---- [+] 762 list: 0x2a6e0e0*4, 0x2a6e488*4, 0x2a6e4d0*1,
0x2a6e518*1...

 7: g      [+] 146 dict of test_backend_ldap.LDAPIdentity:
0x44b36d0...

 8: h ---- [+] 146 dict of test_backend_ldap.LDAPIdentityEnabledEmulation:
0x...
 9: i      [+] 129 dict of test_backend_sql.SqlIdentity: 0x4522b90,
0x50b1290...

If you watch this after running a few more tests, it's the #4 entry where
the number keeps going up, so looking into this more:

(Pdb)
h1.heap()[0].get_rp(depth=1)[4].get_rp()

Reference Pattern by <[dict of]
class>.

 0: _ --- [-] 873 dict (no owner): 0x1aaff60*10, 0x1bf9c90*2,
0x1f7ef10*34...

 1: a      [-] 274 dict (no owner): 0x259c0f0*1, 0x259c420*8,
0x25b5a10*1...

 2: aa ---- [-] 233 dict of
keystone.catalog.backends.templated.TemplatedCata...

 3: a3       [-] 233 keystone.catalog.backends.templated.TemplatedCatalog:
0x...
 4: a4 ------ [-] 60 dict of test_keystoneclient.KcEssex3TestCase:
0x7a20750...
 5: a5         [-] 60 test_keystoneclient.KcEssex3TestCase:
0x76703d0...

 6: a6 -------- [-] 60 dict of nose.case.Test: 0x7a17650...,
0x7a17690

 7: a7           [+] 60 nose.case.Test: 0x6de91d0, 0x6de9210,
0x6de9350...

 8: a5b ------- [-] 60 dict of unittest2.case._TypeEqualityDict:
0x7a20b50...
 9: a5ba         [+] 60 unittest2.case._TypeEqualityDict:
0x6b5bad0...

<Type e.g. '_.more' for
more.>

(Pdb)
h1.heap()[0].get_rp(depth=1)[4].get_rp().more

10: a3b ----- [-] 57 dict of test_keystoneclient.Kc11TestCase:
0x7883150...

11: a3ba       [-] 57 test_keystoneclient.Kc11TestCase: 0x7883150,
0x7b43150...
12: a3baa ----- [-] 57 dict of nose.case.Test: 0x7a389d0,
0x8394d50...

13: a3ba3        [+] 57 nose.case.Test: 0x7a38490, 0x7a385d0,
0x7a389d0...

14: a3bab ----- [-] 57 dict of unittest2.case._TypeEqualityDict:
0x7b63dd0...
15: a3baba       [+] 57 unittest2.case._TypeEqualityDict:
0x7883510...

16: a3c ----- [-] 31 dict of test_keystoneclient.KcMasterTestCase:
0x6dc1490...
17: a3ca       [-] 31 test_keystoneclient.KcMasterTestCase:
0x6aadc90...

18: a3caa ----- [-] 31 dict of nose.case.Test: 0x83225d0,
0x8322610...

19: a3ca3        [+] 31 nose.case.Test: 0x83225d0, 0x8322610,
0x8322890...


The smoking gun here is that the source of the references is nose.case.Test.

So I think that the nose runner is keeping references to the the dicts that
are compared. Also, it looks like nose keeps a reference to the test
instance (along with any objects created in setUp with self.xxx = Xxx()),
but this isn't using so much memory as the test dicts.

- Brant

On Mon, Jun 17, 2013 at 6:06 PM, Monty Taylor <mordred at inaugust.com> wrote:

...


> - unittests should always work on a devstack host inside of a venv
>
> I mean, it's a venv - the fact that you're on a devstack machine is
> irrelvant. Btw- apparently keystone and python-keystone client do not
> work in this context - we should probably sort that out.
>
> ...

>
> Monty
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130619/b9420995/attachment.html>