Open Stack

On 03/27/2015 08:47 PM, Alan Pevec wrote:
> But how come that same recent pyOpenSSL doesn't consume more memory on Ubuntu?

Just to loop back on the final status of this ...

pyOpenSSL 0.14 does seem to use about an order of magnitude more
memory than 0.13 (2mb -> 20mb).  For details see [1].

This is due to the way it now goes through "cryptography" (the
package, not the concept :) which binds to openssl using cffi.  This
ends up parsing a bunch of C to build up the ABI representation, and
it seems pycparser's model of this consumes most of the memory [2].
If that is a bug or not remains to be seen.

Ubuntu doesn't notice this in our CI environment because it comes with
python-openssl 0.13 pre-installed in the image.  Centos started
hitting this when I merged my change to start using as many libraries
from pip as possible.

I have a devstack workaround for centos out (preinstall the package)
[3] and I think a global solution of avoiding it in requirements [4]
(reviews appreciated).

I'm also thinking about how we can better monitor memory usage for
jobs.  Being able to see exactly what change pushed up memory usage by
a large % would have made finding this easier.  We keep some overall
details for devstack runs in a log file, but there is room to do
better.

-i

[1] https://etherpad.openstack.org/p/oom-in-rax-centos7-CI-job
[2] https://github.com/eliben/pycparser/issues/72
[3] https://review.openstack.org/168217
[4] https://review.openstack.org/169596