[openstack-dev] [nova] [Openstack-operators] Profiling nova-conductor and eventlet

Kris G. Lindgren klindgren at godaddy.com
Thu Nov 19 16:29:39 UTC 2015


Calling all Proflers!

I am running into an issue with CPU usage on remote nova-conductor and I am trying to profile it to see where its consuming the most amount of cpu, so that we can investigate further.  The etherpad where we have been working on this issue is located at: https://etherpad.openstack.org/p/remote-conductor-performance

I tired running nova conductor under Cprofiler with 20 workers, however it only saw the main thread.  So I started conducotr with a single worker and cProfiler was able to see more requests.
I ran the following:

  *   python -m cProfile -o conductor-1worker /usr/bin/nova-conductor

  *   ./gprof2dot/gprof2dot.py -f pstats conductor-1worker | dot -Tsvg -o conductor-1worker.svg

  *   SVG output here: http://tempsend.com/5340867576/F283/conductor1worker.svg

The SVG showed that most of the time was spent in multiple different locations but AMQP seemed to be the highest consume of time.  However, I am reading that cProfiler can get confused under eventlet so I am wondering if the cProfiler can be trusted.  I tired using greenletprofiler however, that started to causes errors to be thrown in the conductor logs around connecting to rabbitmq.

So what I am asking is those of you who have experience debugging python with eventlet, what tooling do you use?
___________________________________________________________________
Kris Lindgren
Senior Linux Systems Engineer
GoDaddy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151119/8ed771cc/attachment-0001.html>


More information about the OpenStack-dev mailing list