<div dir="ltr"><br><div class="gmail_extra">I was looking into this with Anita Kuno today.<br><br></div><div class="gmail_extra">Keystone unit tests would fail when she was running on her vm set up for testing. We found that the system was running out of memory. When I run it myself, watching memory usage with top, the nosetester process uses over 800MB. Anita's vm has 1GB total, and she sent me a screenshot showing that the oom killer got it. So it's not that the Keystone unit tests don't work in a venv generally -- you need enough of RAM.<br>
<br></div><div class="gmail_extra">Memory use growing without bound indicates a memory leak, so I started looking into where references were piling up. I was hoping it would be an easy fix where the Keystone test code could be changed to clear a list or something. I haven't looked at python memory leaks before so I might be wrong on all this, but here's my analysis.<br>
<br></div><div class="gmail_extra">The summary is the references are piling up in the nosetests runner as it's storing the results of all the tests, so I don't think there's anything we can do in Keystone code to fix it.<br>
</div><div class="gmail_extra"><br></div><div class="gmail_extra">I made use of heapy <a href="https://pypi.python.org/pypi/guppy/0.1.10">https://pypi.python.org/pypi/guppy/0.1.10</a> to find the heap usage.<br><br></div>
<div class="gmail_extra">After running 872 tests,<br><br>> /usr/lib/python2.7/dist-packages/nose/suite.py(223)run()<br>-> test(orig)<br>(Pdb) p result<br><nose.result.TextTestResult run=869 errors=3 failures=0><br>
<br>hpy.heap() shows the biggest user of the heap is unnamed dicts.<br></div><div class="gmail_extra"><br>(Pdb) h1.heap()<br>Partition of a set of 487786 objects. Total size = 88365600 bytes.<br> Index Count % Size % Cumulative % Kind (class / dict of class)<br>
0 76542 16 27581904 31 27581904 31 dict (no owner)<br> 1 128620 26 11649056 13 39230960 44 str<br> 2 72387 15 6771416 8 46002376 52 tuple<br> 3 3043 1 3189064 4 49191440 56 dict of 0x2b96700<br>
4 21977 5 2615816 3 51807256 59 list<br> 5 20374 4 2444880 3 54252136 61 function<br> 6 739 0 2435656 3 56687792 64 dict of sqlalchemy.dialects.sqlite.base.SQLiteCompiler<br> 7 2579 1 2324312 3 59012104 67 type<br>
8 17378 4 2224384 3 61236488 69 types.CodeType<br> 9 854 0 2123792 2 63360280 72 dict of module<br><899 more rows. Type e.g. '_.more' to view.><br><br></div><div class="gmail_extra">
By running for a while and watching the list change, one sees the "dict (no owner)" line start out at the bottom of the list and climb up to the top as more tests run.<br></div><div class="gmail_extra"><br></div>
<div class="gmail_extra">Digging into the dict data a bit more, one can see what's holding the refs to dicts,<br><br>(Pdb) h1.heap([0].get_rp(depth=1) <br>
<br>Reference Pattern by <[dict of] class>. <br> 0: _ --- [-] 76503 dict (no owner): 0x1a4f150*1673, 0x1a52380*15... <br>
1: a [+] 739 dict of sqlalchemy.dialects.sqlite.base.SQLiteCompiler: 0x... <br> 2: b ---- [+] 514 dict of routes.route.Route: 0x5db1b50..., 0x68f02d0 <br>
3: c [+] 905 dict of nose.case.Test: 0x2a78490, 0x2ad2c90, 0x2ad2e50... <br> 4: d ---- [+] 873 dict (no owner): 0x1aaff60*10, 0x1bf9c90*2, 0x1f7ef10*34... <br>
5: e [S] 157 dict of module: _warnings, errno..., os, sys, zipimport <br> 6: f ---- [+] 762 list: 0x2a6e0e0*4, 0x2a6e488*4, 0x2a6e4d0*1, 0x2a6e518*1... <br>
7: g [+] 146 dict of test_backend_ldap.LDAPIdentity: 0x44b36d0... <br> 8: h ---- [+] 146 dict of test_backend_ldap.LDAPIdentityEnabledEmulation: 0x... <br>
9: i [+] 129 dict of test_backend_sql.SqlIdentity: 0x4522b90, 0x50b1290... <br><br></div><div class="gmail_extra">If you watch this after running a few more tests, it's the #4 entry where the number keeps going up, so looking into this more:<br>
<br>(Pdb) h1.heap()[0].get_rp(depth=1)[4].get_rp() <br>Reference Pattern by <[dict of] class>. <br>
0: _ --- [-] 873 dict (no owner): 0x1aaff60*10, 0x1bf9c90*2, 0x1f7ef10*34... <br> 1: a [-] 274 dict (no owner): 0x259c0f0*1, 0x259c420*8, 0x25b5a10*1... <br>
2: aa ---- [-] 233 dict of keystone.catalog.backends.templated.TemplatedCata... <br> 3: a3 [-] 233 keystone.catalog.backends.templated.TemplatedCatalog: 0x... <br>
4: a4 ------ [-] 60 dict of test_keystoneclient.KcEssex3TestCase: 0x7a20750... <br> 5: a5 [-] 60 test_keystoneclient.KcEssex3TestCase: 0x76703d0... <br>
6: a6 -------- [-] 60 dict of nose.case.Test: 0x7a17650..., 0x7a17690 <br> 7: a7 [+] 60 nose.case.Test: 0x6de91d0, 0x6de9210, 0x6de9350... <br>
8: a5b ------- [-] 60 dict of unittest2.case._TypeEqualityDict: 0x7a20b50... <br> 9: a5ba [+] 60 unittest2.case._TypeEqualityDict: 0x6b5bad0... <br>
<Type e.g. '_.more' for more.> <br>(Pdb) h1.heap()[0].get_rp(depth=1)[4].get_rp().more <br>
10: a3b ----- [-] 57 dict of test_keystoneclient.Kc11TestCase: 0x7883150... <br>11: a3ba [-] 57 test_keystoneclient.Kc11TestCase: 0x7883150, 0x7b43150... <br>
12: a3baa ----- [-] 57 dict of nose.case.Test: 0x7a389d0, 0x8394d50... <br>13: a3ba3 [+] 57 nose.case.Test: 0x7a38490, 0x7a385d0, 0x7a389d0... <br>
14: a3bab ----- [-] 57 dict of unittest2.case._TypeEqualityDict: 0x7b63dd0... <br>15: a3baba [+] 57 unittest2.case._TypeEqualityDict: 0x7883510... <br>
16: a3c ----- [-] 31 dict of test_keystoneclient.KcMasterTestCase: 0x6dc1490... <br>17: a3ca [-] 31 test_keystoneclient.KcMasterTestCase: 0x6aadc90... <br>
18: a3caa ----- [-] 31 dict of nose.case.Test: 0x83225d0, 0x8322610... <br>19: a3ca3 [+] 31 nose.case.Test: 0x83225d0, 0x8322610, 0x8322890... <br>
<br></div><div class="gmail_extra">The smoking gun here is that the source of the references is nose.case.Test.<br></div><div class="gmail_extra"><br></div><div class="gmail_extra">So I think that the nose runner is keeping references to the the dicts that are compared. Also, it looks like nose keeps a reference to the test instance (along with any objects created in setUp with self.xxx = Xxx()), but this isn't using so much memory as the test dicts.<br>
</div><div class="gmail_extra"><br></div><div class="gmail_extra">- Brant<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 17, 2013 at 6:06 PM, Monty Taylor <span dir="ltr"><<a href="mailto:mordred@inaugust.com" target="_blank">mordred@inaugust.com</a>></span> wrote:<br>
<div><br>...<br> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">- unittests should always work on a devstack host inside of a venv<br>
<br>
I mean, it's a venv - the fact that you're on a devstack machine is<br>
irrelvant. Btw- apparently keystone and python-keystone client do not<br>
work in this context - we should probably sort that out.<br>
<br></blockquote><div>... <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Monty<br>
<br></blockquote></div></div></div>