<div dir="ltr">What I don't understand is why the OOM killer is being invoked when there is almost no swap space being used at all. Check out the memory output when it's killed:<div><br></div><div><a href="http://logs.openstack.org/59/382659/26/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/7de01d0/logs/syslog.txt.gz#_Jan_11_15_54_36">http://logs.openstack.org/59/382659/26/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/7de01d0/logs/syslog.txt.gz#_Jan_11_15_54_36</a><br></div><div><br></div><div>"Jan 11 15:54:36 ubuntu-xenial-rax-ord-6599274 kernel: Free swap  = 7994832kB</div><div>Jan 11 15:54:36 ubuntu-xenial-rax-ord-6599274 kernel: Total swap = 7999020kB"</div><div><br></div><div>Do we have something set that is effectively disabling the usage of swap space?</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 18, 2017 at 4:13 PM, Joe Gordon <span dir="ltr"><<a href="mailto:joe.gordon0@gmail.com" target="_blank">joe.gordon0@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Thu, Jan 19, 2017 at 10:27 AM, Matt Riedemann <span dir="ltr"><<a href="mailto:mriedem@linux.vnet.ibm.com" target="_blank">mriedem@linux.vnet.ibm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class="m_-2476877688946697426gmail-">On 1/18/2017 4:53 AM, Jens Rosenboom wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
To me it looks like the times of 2G are long gone, Nova is using<br>
almost 2G all by itself. And 8G may be getting tight if additional<br>
stuff like Ceph is being added.<br>
<br>
</blockquote>
<br></span>
I'm not really surprised at all about Nova being a memory hog with the versioned object stuff we have which does it's own nesting of objects.<br>
<br>
What tools to people use to be able to profile the memory usage by the types of objects in memory while this is running?</blockquote><div><br></div></span><div>objgraph and guppy/heapy</div><div><br></div><div><a href="http://smira.ru/wp-content/uploads/2011/08/heapy.html" target="_blank">http://smira.ru/wp-content/<wbr>uploads/2011/08/heapy.html</a><br></div><div> </div><div><a href="https://www.huyng.com/posts/python-performance-analysis" target="_blank">https://www.huyng.com/posts/<wbr>python-performance-analysis</a><br></div><div><br></div><div>You can also use gc.get_objects() (<a href="https://docs.python.org/2/library/gc.html#gc.get_objects" target="_blank">https://docs.python.org/2/<wbr>library/gc.html#gc.get_objects</a><wbr>) to get a list of all objects in memory and go from there.</div><div><br></div><div>Slots (<a href="https://docs.python.org/2/reference/datamodel.html#slots" target="_blank">https://docs.python.org/2/<wbr>reference/datamodel.html#slots</a><wbr>) are useful for reducing the memory usage of objects.</div><span class=""><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class="m_-2476877688946697426gmail-HOEnZb"><font color="#888888"><br>
<br>
-- <br>
<br>
Thanks,<br>
<br>
Matt Riedemann</font></span><div class="m_-2476877688946697426gmail-HOEnZb"><div class="m_-2476877688946697426gmail-h5"><br>
<br>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
</div></div></blockquote></span></div><br></div></div>
<br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br></blockquote></div><br></div>