<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
Calling all Proflers!</div>
<div>
<div id="">
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri"><span class="Apple-style-span" style="font-size: 14px;"><br>
</span></font></font></div>
<div>I<font class="Apple-style-span" color="#000000"><font face="Calibri,sans-serif" style="font-family: Calibri;"> am running into an issue with CPU </font>usage<font face="Calibri" style="font-family: Calibri, sans-serif;"> on remote nova-conductor and </font><font face="Calibri,sans-serif">I</font><font face="Calibri"><font face="Calibri,sans-serif"> am
trying to profile it to see where its consuming </font>the<font face="Calibri,sans-serif"> most amount of cpu, so that we can investigate further. The etherpad where we have been working on this issue is located at: </font></font></font><a href="https://etherpad.openstack.org/p/remote-conductor-performance">https://etherpad.openstack.org/p/remote-conductor-performance</a></div>
<div><br>
</div>
<div>I tired running nova conductor under Cprofiler with 20 workers, however it only saw the main thread. So I started conducotr with a single worker and cProfiler was able to see more requests.</div>
<div>I ran the following: </div>
<div id="magicdomid2931" class="ace-line" style="margin: 0px; padding: 0px; line-height: 16px; widows: 1;">
<ul class="list-bullet2" style="margin: 0px 0px 0px 3em; padding: 0px; list-style-type: circle;">
<li style="margin: 0px; padding: 0px;"><span class="author-a-z65zfyz71zz65zz66zwrz85zdz65z7bbfz67z" style="margin: 0px; padding: 1px 0px; cursor: auto;">python -m cProfile -o conductor-1worker /usr/bin/nova-conductor</span></li></ul>
</div>
<div id="magicdomid2932" class="ace-line" style="margin: 0px; padding: 0px; line-height: 16px; widows: 1;">
<ul class="list-bullet2" style="margin: 0px 0px 0px 3em; padding: 0px; list-style-type: circle;">
<li style="margin: 0px; padding: 0px;"><span class="author-a-z65zfyz71zz65zz66zwrz85zdz65z7bbfz67z" style="margin: 0px; padding: 1px 0px; cursor: auto;">./gprof2dot/gprof2dot.py -f pstats conductor-1worker | dot -Tsvg -o conductor-1worker.svg</span></li></ul>
</div>
<div id="magicdomid2933" class="ace-line" style="margin: 0px; padding: 0px; widows: 1;">
<ul class="list-bullet2" style="line-height: 16px; margin: 0px 0px 0px 3em; padding: 0px; list-style-type: circle;">
<li style="margin: 0px; padding: 0px;"><span class="author-a-z65zfyz71zz65zz66zwrz85zdz65z7bbfz67z" style="margin: 0px; padding: 1px 0px; cursor: auto;">SVG output here: </span><span class="author-a-z65zfyz71zz65zz66zwrz85zdz65z7bbfz67z url" style="margin: 0px; padding: 1px 0px; cursor: auto;"><a href="http://tempsend.com/5340867576/F283/conductor1worker.svg" style="margin: 0px; padding: 0px; white-space: pre-wrap; cursor: pointer !important;">http://tempsend.com/5340867576/F283/conductor1worker.svg</a></span></li></ul>
<div style="line-height: 16px;"><span style="font-style: normal; font-weight: normal; text-decoration: underline;"><br>
</span></div>
<div><span style="line-height: 16px;">The SVG showed that most of the time was spent in multiple different locations but AMQP seemed to be the highest consume of time. However, I am reading that cProfiler can get confused under eventlet so I am wondering if
the cProfiler can be trusted. I tired using greenletprofiler however, that started to causes errors to be thrown in the conductor logs around connecting to rabbitmq. </span></div>
<div><span style="line-height: 16px;"><br>
</span></div>
<div><span style="line-height: 16px;">So what I am asking is those of you who have experience debugging python with eventlet, what tooling do you use?</span></div>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri">___________________________________________________________________</font></font></div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri">Kris Lindgren</font></font></div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri">Senior Linux Systems Engineer</font></font></div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri">GoDaddy</font></font></div>
</div>
</div>
</body>
</html>