Open Stack

On 03/26/2015 04:07 PM, Ian Wienand wrote:
> See [1] for some more details; but the short story is that the various
> swift processes -- even just sitting around freshly installed from
> devstack before anything happens -- take up twice as much space on
> centos as ubuntu
> 
> --- swift (% total system memory) ---
> ubuntu  6.6%
> centos  12%

So after more investigation, it turns out that pyOpenSSL has rewritten
itself in python; necessitating dependencies on the "cryptography"
package and cffi & pycparser [1].  Examining the heap shows where the
memory has gone missing :

Partition of a set of 205366 objects. Total size = 30969040 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  67041  33  5712560  18   5712560  18 str
     1  10260   5  2872800   9   8585360  28 dict of pycparser.plyparser.Coord
     2  27765  14  2367552   8  10952912  35 tuple
     3   1215   1  2246760   7  13199672  43 dict (no owner)
     4   1882   1  1972336   6  15172008  49 dict of pycparser.c_ast.Decl
     5  16085   8  1736232   6  16908240  55 list
     6    360   0  1135296   4  18043536  58 dict of module
     7   4041   2  1131480   4  19175016  62 dict of pycparser.c_ast.TypeDecl
     8   4021   2  1125880   4  20300896  66 dict of pycparser.c_ast.IdentifierType
     9   6984   3   893952   3  21194848  68 types.CodeType
<413 more rows. Type e.g. '_.more' to view.>

If I reinstall the packaged version of pyOpenSSL, all that drops out
and we're back to a more reasonable usage

Partition of a set of 95591 objects. Total size = 12500080 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  45837  48  3971040  32   3971040  32 str
     1  22843  24  1943416  16   5914456  47 tuple
     2    298   0   978160   8   6892616  55 dict of module
     3   6065   6   776320   6   7668936  61 types.CodeType
     4    551   1   742184   6   8411120  67 dict (no owner)
     5    805   1   725520   6   9136640  73 type
     6   5876   6   705120   6   9841760  79 function
     7    805   1   666232   5  10507992  84 dict of type
     8    289   0   279832   2  10787824  86 dict of class
     9    152   0   159296   1  10947120  88 dict of pkg_resources.Distribution
<310 more rows. Type e.g. '_.more' to view.>

The end result of this is that swift-* processes go from consuming
about 6% of a CI VM's 8gb to 12%.  This 500mb is enough to push the
host into OOM when tempest gets busy.  For more see [2].  a workaround is [3]

I'll spend a bit more time on this -- I haven't determined if it's
centos or swift specific yet -- but in the mean-time, beware of
recent pyOpenSSL

-i

[1] https://github.com/pyca/pyopenssl/commit/fd193a2f9dd8be80d9f42d8dd8068de5f5ac5e67 
[2] https://etherpad.openstack.org/p/oom-in-rax-centos7-CI-job
[3] https://review.openstack.org/#/c/168217/