<div dir="ltr"><div>Thanks for doing this. A couple of questions:</div><div><br></div>What were your rootwrap settings when running these tests? Did you just have it calling sudo directly?<div><br></div><div>Also, you mention that this is only ~10% of the time spent during flow reconfiguration. What other areas are eating up so much time? </div><div><br></div><div>Cheers,</div><div>Kevin Benton</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jan 17, 2016 at 10:12 PM, IWAMOTO Toshihiro <span dir="ltr"><<a href="mailto:iwamoto@valinux.co.jp" target="_blank">iwamoto@valinux.co.jp</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I'm sending out this mail to share the finding and discuss how to<br>
improve with those interested in neutron ovs performance.<br>
<br>
TL;DR: The native of_interface code, which has been merged recently<br>
and isn't default, seems to consume less CPU time but gives a mixed<br>
result. I'm looking into this for improvement.<br>
<br>
* Introduction<br>
<br>
With an ML2+ovs Neutron configuration, openflow rule modification<br>
happens often and is somewhat a heavy operation as it involves<br>
exec() of the ovs-ofctl command.<br>
<br>
The native of_interface driver doesn't use the ovs-ofctl command and<br>
should have less performance impact on the system. This document<br>
tries to confirm this hypothesis.<br>
<br>
<br>
* Method<br>
<br>
In order to focus on openflow rule operation time and avoid noise from<br>
other operations (VM boot-up, etc.), neutron-openvswitch-agent was<br>
restarted and the time it took to reconfigure the flows was measured.<br>
<br>
1. Use devstack to start a test environment. As debug logs generate<br>
considable amount of load, ENABLE_DEBUG_LOG_LEVEL was set to false.<br>
2. Apply <a href="https://review.openstack.org/#/c/267905/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/267905/</a> to enable<br>
measurement of flow reconfiguration times.<br>
3. Boot 80 m1.nano instances. In my setup, this generates 404 br-int<br>
flows. If you have >16G RAM, more could be booted.<br>
4. Stop neutron-openvswitch-agent and restart with --run-once arg.<br>
Use time, oprofile, and python's cProfile (use --profile arg) to<br>
collect data.<br>
<br>
* Results<br>
<br>
Execution time (averages of 3 runs):<br>
<br>
native 28.3s user 2.9s sys 0.4s<br>
ovs-ofctl 25.7s user 2.2s sys 0.3s<br>
<br>
ovs-ofctl runs faster and seems to use less CPU, but the above doesn't<br>
count in execution time of ovs-ofctl.<br>
<br>
Oprofile data collected by running "operf -s -t" contain the<br>
information.<br>
<br>
With of_interface=native config, "opreport tgid:<pid of ovs agent>" shows:<br>
<br>
samples| %|<br>
------------------<br>
87408 100.000 python2.7<br>
CPU_CLK_UNHALT...|<br>
samples| %|<br>
------------------<br>
69160 79.1232 python2.7<br>
8416 9.6284 vmlinux-3.13.0-24-generic<br>
<br>
and "opreport --merge tgid" doesn't show ovs-ofctl.<br>
<br>
With of_interface=ovs-ofctl, "opreport tgid:<pid of ovs agent>" shows:<br>
<br>
samples| %|<br>
------------------<br>
62771 100.000 python2.7<br>
CPU_CLK_UNHALT...|<br>
samples| %|<br>
------------------<br>
49418 78.7274 python2.7<br>
6483 10.3280 vmlinux-3.13.0-24-generic<br>
<br>
and "opreport --merge tgid" shows CPU consumption by ovs-ofctl<br>
<br>
35774 3.5979 ovs-ofctl<br>
CPU_CLK_UNHALT...|<br>
samples| %|<br>
------------------<br>
28219 78.8813 vmlinux-3.13.0-24-generic<br>
3487 9.7473 <a href="http://ld-2.19.so" rel="noreferrer" target="_blank">ld-2.19.so</a><br>
2301 6.4320 ovs-ofctl<br>
<br>
Comparing 87408 (native python) with 62771+35774, the native<br>
of_interface uses 0.4s less CPU time overall.<br>
<br>
* Conclusion and future steps<br>
<br>
The native of_interface uses slightly less CPU time but takes longer<br>
time to complete a flow reconfiguration after an agent restart.<br>
<br>
As an OVS agent accounts for only 1/10th of total CPU usage during a<br>
flow reconfiguration (data not shown), there may be other areas for<br>
improvement.<br>
<br>
The cProfile Python module gives more fine grained data, but no<br>
apparent performance bottleneck was found. The data show more<br>
eventlet context switches with the native of_interface, which is due<br>
to how the native of_interface is written. I'm looking into for<br>
improving CPU usage and latency.<br>
<br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div>Kevin Benton</div></div>
</div>