Open Stack

Hi,

We've been having an issue with Centos 7 jobs and the host running
out of memory.  After some investigation; one likely angle seems to be
the memory usage by swift.

See [1] for some more details; but the short story is that the various
swift processes -- even just sitting around freshly installed from
devstack before anything happens -- take up twice as much space on
centos as ubuntu

--- swift (% total system memory) ---
ubuntu  6.6%
centos  12%

In general memory usage is higher on centos, but this one is an
outlier.  Unfortunately, the main difference between the two appears
to be heap allocations (see comments in [1]) which doesn't give a lot
of clues.

The end result is that the host ends up just plain running out of
memory; the OOM killer kicks in and then everything starts
collapsing. I had the host sending me telemetry while it was running;
the last entry before things fell over was [2] and we can see that
it's not just one thing that comes along and sucks up memory, but
death by a thousand cuts.  I think the Centos 7 job is struggling to
fit into the 8gb available so we're susceptible to finding memory
issues first.

Any swift people have some ideas on this?

Thanks

-i

[1] https://etherpad.openstack.org/p/oom-in-rax-centos7-CI-job
[2] http://paste.openstack.org/show/196769/