Open Stack

It would strike me as very odd if nose was the culprit (inherently vs. 
just poorly written tests). Before nova switched to testr we had over 
5000 tests, which is 4x what's in keystone right now.

Also, tempest has > 1000 tests running in nose, and very clearly can 
function on a devstack gate node. And we allocate lots of servers to 
boot. :)

	-Sean

On 06/19/2013 07:22 PM, Brant Knudson wrote:
>
> I was looking into this with Anita Kuno today.
>
> Keystone unit tests would fail when she was running on her vm set up for
> testing. We found that the system was running out of memory. When I run
> it myself, watching memory usage with top, the nosetester process uses
> over 800MB. Anita's vm has 1GB total, and she sent me a screenshot
> showing that the oom killer got it. So it's not that the Keystone unit
> tests don't work in a venv generally -- you need enough of RAM.
>
> Memory use growing without bound indicates a memory leak, so I started
> looking into where references were piling up. I was hoping it would be
> an easy fix where the Keystone test code could be changed to clear a
> list or something. I haven't looked at python memory leaks before so I
> might be wrong on all this, but here's my analysis.
>
> The summary is the references are piling up in the nosetests runner as
> it's storing the results of all the tests, so I don't think there's
> anything we can do in Keystone code to fix it.
>
> I made use of heapy https://pypi.python.org/pypi/guppy/0.1.10 to find
> the heap usage.
>
> After running 872 tests,
>
>  > /usr/lib/python2.7/dist-packages/nose/suite.py(223)run()
> -> test(orig)
> (Pdb) p result
> <nose.result.TextTestResult run=869 errors=3 failures=0>
>
> hpy.heap() shows the biggest user of the heap is unnamed dicts.
>
> (Pdb) h1.heap()
> Partition of a set of 487786 objects. Total size = 88365600 bytes.
>   Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
>       0  76542  16 27581904  31  27581904  31 dict (no owner)
>       1 128620  26 11649056  13  39230960  44 str
>       2  72387  15  6771416   8  46002376  52 tuple
>       3   3043   1  3189064   4  49191440  56 dict of 0x2b96700
>       4  21977   5  2615816   3  51807256  59 list
>       5  20374   4  2444880   3  54252136  61 function
>       6    739   0  2435656   3  56687792  64 dict of
> sqlalchemy.dialects.sqlite.base.SQLiteCompiler
>       7   2579   1  2324312   3  59012104  67 type
>       8  17378   4  2224384   3  61236488  69 types.CodeType
>       9    854   0  2123792   2  63360280  72 dict of module
> <899 more rows. Type e.g. '_.more' to view.>
>
> By running for a while and watching the list change, one sees the "dict
> (no owner)" line start out at the bottom of the list and climb up to the
> top as more tests run.
>
> Digging into the dict data a bit more, one can see what's holding the
> refs to dicts,
>
> (Pdb) h1.heap([0].get_rp(depth=1)
>
> Reference Pattern by <[dict of] class>.
>   0: _ --- [-] 76503 dict (no owner): 0x1a4f150*1673, 0x1a52380*15...
>   1: a      [+] 739 dict of
> sqlalchemy.dialects.sqlite.base.SQLiteCompiler: 0x...
>   2: b ---- [+] 514 dict of routes.route.Route: 0x5db1b50..., 0x68f02d0
>   3: c      [+] 905 dict of nose.case.Test: 0x2a78490, 0x2ad2c90,
> 0x2ad2e50...
>   4: d ---- [+] 873 dict (no owner): 0x1aaff60*10, 0x1bf9c90*2,
> 0x1f7ef10*34...
>   5: e      [S] 157 dict of module: _warnings, errno..., os, sys, zipimport
>   6: f ---- [+] 762 list: 0x2a6e0e0*4, 0x2a6e488*4, 0x2a6e4d0*1,
> 0x2a6e518*1...
>   7: g      [+] 146 dict of test_backend_ldap.LDAPIdentity: 0x44b36d0...
>   8: h ---- [+] 146 dict of
> test_backend_ldap.LDAPIdentityEnabledEmulation: 0x...
>   9: i      [+] 129 dict of test_backend_sql.SqlIdentity: 0x4522b90,
> 0x50b1290...
>
> If you watch this after running a few more tests, it's the #4 entry
> where the number keeps going up, so looking into this more:
>
> (Pdb) h1.heap()[0].get_rp(depth=1)[4].get_rp()
> Reference Pattern by <[dict of] class>.
>   0: _ --- [-] 873 dict (no owner): 0x1aaff60*10, 0x1bf9c90*2,
> 0x1f7ef10*34...
>   1: a      [-] 274 dict (no owner): 0x259c0f0*1, 0x259c420*8,
> 0x25b5a10*1...
>   2: aa ---- [-] 233 dict of
> keystone.catalog.backends.templated.TemplatedCata...
>   3: a3       [-] 233
> keystone.catalog.backends.templated.TemplatedCatalog: 0x...
>   4: a4 ------ [-] 60 dict of test_keystoneclient.KcEssex3TestCase:
> 0x7a20750...
>   5: a5         [-] 60 test_keystoneclient.KcEssex3TestCase: 0x76703d0...
>   6: a6 -------- [-] 60 dict of nose.case.Test: 0x7a17650..., 0x7a17690
>   7: a7           [+] 60 nose.case.Test: 0x6de91d0, 0x6de9210, 0x6de9350...
>   8: a5b ------- [-] 60 dict of unittest2.case._TypeEqualityDict:
> 0x7a20b50...
>   9: a5ba         [+] 60 unittest2.case._TypeEqualityDict: 0x6b5bad0...
> <Type e.g. '_.more' for more.>
> (Pdb) h1.heap()[0].get_rp(depth=1)[4].get_rp().more
> 10: a3b ----- [-] 57 dict of test_keystoneclient.Kc11TestCase: 0x7883150...
> 11: a3ba       [-] 57 test_keystoneclient.Kc11TestCase: 0x7883150,
> 0x7b43150...
> 12: a3baa ----- [-] 57 dict of nose.case.Test: 0x7a389d0, 0x8394d50...
> 13: a3ba3        [+] 57 nose.case.Test: 0x7a38490, 0x7a385d0, 0x7a389d0...
> 14: a3bab ----- [-] 57 dict of unittest2.case._TypeEqualityDict:
> 0x7b63dd0...
> 15: a3baba       [+] 57 unittest2.case._TypeEqualityDict: 0x7883510...
> 16: a3c ----- [-] 31 dict of test_keystoneclient.KcMasterTestCase:
> 0x6dc1490...
> 17: a3ca       [-] 31 test_keystoneclient.KcMasterTestCase: 0x6aadc90...
> 18: a3caa ----- [-] 31 dict of nose.case.Test: 0x83225d0, 0x8322610...
> 19: a3ca3        [+] 31 nose.case.Test: 0x83225d0, 0x8322610, 0x8322890...
>
> The smoking gun here is that the source of the references is nose.case.Test.
>
> So I think that the nose runner is keeping references to the the dicts
> that are compared. Also, it looks like nose keeps a reference to the
> test instance (along with any objects created in setUp with self.xxx =
> Xxx()), but this isn't using so much memory as the test dicts.
>
> - Brant
>
> On Mon, Jun 17, 2013 at 6:06 PM, Monty Taylor <mordred at inaugust.com
> <mailto:mordred at inaugust.com>> wrote:
>
> ...
>
>     - unittests should always work on a devstack host inside of a venv
>
>     I mean, it's a venv - the fact that you're on a devstack machine is
>     irrelvant. Btw- apparently keystone and python-keystone client do not
>     work in this context - we should probably sort that out.
>
> ...
>
>
>     Monty
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


-- 
Sean Dague
http://dague.net