What I don't understand is why the OOM killer is being invoked when there is almost no swap space being used at all. Check out the memory output when it's killed: http://logs.openstack.org/59/382659/26/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/7de01d0/logs/syslog.txt.gz#_Jan_11_15_54_36 "Jan 11 15:54:36 ubuntu-xenial-rax-ord-6599274 kernel: Free swap = 7994832kB Jan 11 15:54:36 ubuntu-xenial-rax-ord-6599274 kernel: Total swap = 7999020kB" Do we have something set that is effectively disabling the usage of swap space? On Wed, Jan 18, 2017 at 4:13 PM, Joe Gordon <joe.gordon0 at gmail.com> wrote: > > > On Thu, Jan 19, 2017 at 10:27 AM, Matt Riedemann < > mriedem at linux.vnet.ibm.com> wrote: > >> On 1/18/2017 4:53 AM, Jens Rosenboom wrote: >> >>> To me it looks like the times of 2G are long gone, Nova is using >>> almost 2G all by itself. And 8G may be getting tight if additional >>> stuff like Ceph is being added. >>> >>> >> I'm not really surprised at all about Nova being a memory hog with the >> versioned object stuff we have which does it's own nesting of objects. >> >> What tools to people use to be able to profile the memory usage by the >> types of objects in memory while this is running? > > > objgraph and guppy/heapy > > http://smira.ru/wp-content/uploads/2011/08/heapy.html > > https://www.huyng.com/posts/python-performance-analysis > > You can also use gc.get_objects() (https://docs.python.org/2/ > library/gc.html#gc.get_objects) to get a list of all objects in memory > and go from there. > > Slots (https://docs.python.org/2/reference/datamodel.html#slots) are > useful for reducing the memory usage of objects. > > >> >> -- >> >> Thanks, >> >> Matt Riedemann >> >> >> >> ____________________________________________________________ >> ______________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscrib >> e >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170123/a9e483dc/attachment.html>