[openstack-dev] [nova] [Openstack-operators] Profiling nova-conductor and eventlet

Joshua Harlow harlowja at fastmail.com
Thu Nov 19 17:18:18 UTC 2015


Kris and I also experimented with:

https://gist.github.com/harlowja/d2b5fc5282c76f10bc3f (which is like the 
entrypoint to cProfile but instead using the eventlet profiler module 
instead), that didn't appear to go so well...

So any thoughts would be great here (especially around how to profile 
with eventlet, aka, with something that understands that greenthreads 
are being used and takes that into account).

We also tried https://pypi.python.org/pypi/GreenletProfiler but that 
didn't seem to go so well either.

Any success stories from people?

Kris G. Lindgren wrote:
> Calling all Proflers!
>
> Iam running into an issue with CPU usageon remote nova-conductor and Iam
> trying to profile it to see where its consuming themost amount of cpu,
> so that we can investigate further. The etherpad where we have been
> working on this issue is located at:
> https://etherpad.openstack.org/p/remote-conductor-performance
>
> I tired running nova conductor under Cprofiler with 20 workers, however
> it only saw the main thread. So I started conducotr with a single worker
> and cProfiler was able to see more requests.
> I ran the following:
>
>   * python -m cProfile -o conductor-1worker /usr/bin/nova-conductor
>
>   * ./gprof2dot/gprof2dot.py -f pstats conductor-1worker | dot -Tsvg -o
>     conductor-1worker.svg
>
>   * SVG output here:
>     http://tempsend.com/5340867576/F283/conductor1worker.svg
>
>
> The SVG showed that most of the time was spent in multiple different
> locations but AMQP seemed to be the highest consume of time. However, I
> am reading that cProfiler can get confused under eventlet so I am
> wondering if the cProfiler can be trusted. I tired using
> greenletprofiler however, that started to causes errors to be thrown in
> the conductor logs around connecting to rabbitmq.
>
> So what I am asking is those of you who have experience debugging python
> with eventlet, what tooling do you use?
> ___________________________________________________________________
> Kris Lindgren
> Senior Linux Systems Engineer
> GoDaddy
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



More information about the OpenStack-dev mailing list