<div dir="ltr"><div dir="ltr">Hey Chris, thanks for sharing this :)</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 2, 2020 at 3:30 PM Apsey, Christopher <<a href="mailto:CAPSEY@augusta.edu">CAPSEY@augusta.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_1591513387041266765WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">All,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Just wanted to loop back here and give an update.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">For reference, [1] (blue means successful action, red means failed action) is the result we got when booting 5000 instances in rally [2] before the Red Hat OVN
devs poked around inside our environment, and [3] is the result after. The differences are obviously pretty significant. I think the biggest change was setting metadata_workers = 2 in neutron_ovn_metadata_agent.ini on the compute nodes per
<a href="https://bugs.launchpad.net/neutron/+bug/1893656" target="_blank">https://bugs.launchpad.net/neutron/+bug/1893656</a>. We have 64C/128T on all compute nodes, so the default neutron calculation of scaling metadata workers based on available cores created 900+ connections
to the southbound db at idle; after the control plane got loaded up it just quit around 2500 instances (my guess is it hit the open file limit, although I don’t think increasing it would have made it better for much longer since the number of connections were
increasing exponentially). Capping the number of metadata workers decreased open southbound connections by 90%. Even more telling was that rally was able to successfully clean up after itself after we made that change, whereas previously it wasn’t even able
to successfully tear down any of the instances that were made, indicating that the control plane was completely toast.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Note that the choppiness towards the end of [3] had nothing to do with OVN – our compute nodes had a loadavg approaching 1000 at that point, so they were just
starved for cpu cycles. This would have scaled even better with additional compute nodes.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">The other piece was RAFT. Currently, RDO is shipping with ovs 2.12, but 2.13 has a bunch of RAFT fixes in it that improve stability and knock out some bugs.
We were having issues with chassis registration on 2.12, but after using the 2.13 package from cbs, all those issues went away.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Big thanks to the great people at Red Hat on the cc line for volunteering their valuable time to take a look.</span></p></div></div></blockquote><div><br></div><div>Happy to help, it was fun :) Thanks to you for all the details that made it easier to debug</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_1591513387041266765WordSection1"><p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">I’m now significantly more comfortable with defaulting to OVN as the backend of choice as the performance delta is now gone. That said, should the community
consider dropping linuxbridge as the backend in the official upstream docs and jump straight to OVN rather than ml2/OVS? I think that would increase the test base and help shine light on other issues as time goes on. My org can devote some time to doing
this work if the community agrees that it’s the right action to take.</span></p></div></div></blockquote><div><br></div><div>++!! </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_1591513387041266765WordSection1"><p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Hope that’s helpful!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">[1]
<a href="https://ibb.co/GTjZP2y" target="_blank">https://ibb.co/GTjZP2y</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">[2]
</span><a href="https://pastebin.com/5pEDZ7dY" target="_blank">https://pastebin.com/5pEDZ7dY</a><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">[3]
<a href="https://ibb.co/pfB9KTV" target="_blank">https://ibb.co/pfB9KTV</a></span></p></div></div></blockquote><div><br></div><div>Do you have some baseline to compare against? Also I'm curious to see if you pulled results with and without raft :) </div><div><br></div><div>Thanks once again!</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_1591513387041266765WordSection1"><p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Helvetica LT Std Cond Light";color:black">Chris Apsey<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Helvetica LT Std Black";color:rgb(193,215,46)">GEORGIA CYBER CENTER<u></u><u></u></span></b></p>
</div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:Calibri,sans-serif"> Apsey, Christopher
<br>
<b>Sent:</b> Thursday, August 27, 2020 11:33 AM<br>
<b>To:</b> Assaf Muller <<a href="mailto:amuller@redhat.com" target="_blank">amuller@redhat.com</a>><br>
<b>Cc:</b> <a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>; Lucas Alvares Gomes Martins <<a href="mailto:lmartins@redhat.com" target="_blank">lmartins@redhat.com</a>>; Jakub Libosvar <<a href="mailto:jlibosva@redhat.com" target="_blank">jlibosva@redhat.com</a>>; Daniel Alvarez Sanchez <<a href="mailto:dalvarez@redhat.com" target="_blank">dalvarez@redhat.com</a>><br>
<b>Subject:</b> RE: [EXTERNAL] Re: [neutron][ovn] OVN Performance<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Assaf,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">We can absolutely support engineering poking around in our environment (and possibly an even larger one at my previous employer that was experiencing similar
issues during testing). We can take this offline so we don’t spam the mailing list.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Just let me know how to proceed,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thanks!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Helvetica LT Std Cond Light";color:black">Chris Apsey<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Helvetica LT Std Black";color:rgb(193,215,46)">GEORGIA CYBER CENTER<u></u><u></u></span></b></p>
</div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:Calibri,sans-serif"> Assaf Muller <</span><a href="mailto:amuller@redhat.com" target="_blank"><span style="font-size:11pt;font-family:Calibri,sans-serif">amuller@redhat.com</span></a><span style="font-size:11pt;font-family:Calibri,sans-serif">>
<br>
<b>Sent:</b> Thursday, August 27, 2020 11:18 AM<br>
<b>To:</b> Apsey, Christopher <</span><a href="mailto:CAPSEY@augusta.edu" target="_blank"><span style="font-size:11pt;font-family:Calibri,sans-serif">CAPSEY@augusta.edu</span></a><span style="font-size:11pt;font-family:Calibri,sans-serif">><br>
<b>Cc:</b> </span><a href="mailto:openstack-discuss@lists.openstack.org" target="_blank"><span style="font-size:11pt;font-family:Calibri,sans-serif">openstack-discuss@lists.openstack.org</span></a><span style="font-size:11pt;font-family:Calibri,sans-serif">; Lucas
Alvares Gomes Martins <</span><a href="mailto:lmartins@redhat.com" target="_blank"><span style="font-size:11pt;font-family:Calibri,sans-serif">lmartins@redhat.com</span></a><span style="font-size:11pt;font-family:Calibri,sans-serif">>; Jakub Libosvar <</span><a href="mailto:jlibosva@redhat.com" target="_blank"><span style="font-size:11pt;font-family:Calibri,sans-serif">jlibosva@redhat.com</span></a><span style="font-size:11pt;font-family:Calibri,sans-serif">>;
Daniel Alvarez Sanchez <</span><a href="mailto:dalvarez@redhat.com" target="_blank"><span style="font-size:11pt;font-family:Calibri,sans-serif">dalvarez@redhat.com</span></a><span style="font-size:11pt;font-family:Calibri,sans-serif">><br>
<b>Subject:</b> [EXTERNAL] Re: [neutron][ovn] OVN Performance<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">CAUTION: EXTERNAL SENDER This email originated from an external source. Please exercise caution before opening attachments, clicking links, replying, or providing information to the sender. If you believe it to be fraudulent, contact the
AU Cybersecurity Hotline at 72-CYBER (2-9237 / 706-722-9237) or <a href="mailto:72CYBER@augusta.edu" target="_blank">
72CYBER@augusta.edu</a><br>
<br>
The most efficient way about this is to give one or more of the<br>
Engineers working on OpenStack OVN upstream (I've added a few to this<br>
thread) temporary access to an environment that can reproduce issues<br>
you're seeing, we could then document the issues and work towards<br>
solutions. If that's not possible, if you could provide reproducer<br>
scripts, or alternatively sharpen the reproduction method, we'll take<br>
a look. What you've described is not something that's 'acceptable',<br>
OVN should definitely not scale worse than Neutron with the Linux<br>
Bridge agent. It's possible that the particular issues you ran in to<br>
is something that we've already seen internally at Red Hat, or with<br>
our customers, and we're already working on fixes in future versions<br>
of OVN - I can't tell you until you elaborate on the details of the<br>
issues you're seeing. In any case, the upstream community is committed<br>
to improving OVN scale and fixing scale issues as they pop up.<br>
Coincidentally, Red Hat scale engineers just published an article [1]<br>
about work they've done to scale RH-OSP 16.1 (== OpenStack Train on<br>
CentOS 8, with OVN 2.13 and TripleO) to 700 compute nodes.<br>
<br>
[1] <a href="https://www.redhat.com/en/blog/scaling-red-hat-openstack-platform-161-more-700-nodes?source=bloglisting" target="_blank">https://www.redhat.com/en/blog/scaling-red-hat-openstack-platform-161-more-700-nodes?source=bloglisting</a><br>
<br>
On Thu, Aug 27, 2020 at 10:44 AM Apsey, Christopher <<a href="mailto:CAPSEY@augusta.edu" target="_blank">CAPSEY@augusta.edu</a>> wrote:<br>
><br>
> All,<br>
><br>
><br>
><br>
> I know that OVN is going to become the default neutron backend at some point and displace linuxbridge as the default configuration option in the docs, but we have noticed a pretty significant performance disparity between OVN and linuxbridge on identical
hardware over the past year or so in a few different environments[1]. I know that example is unscientific, but similar results have been borne out in many different scenarios from what we have observed. There are three main problems from what we see:<br>
><br>
><br>
><br>
> 1. OVN does not handle large concurrent requests as well as linuxbridge. Additionally, linuxbridge concurrent capacity grows (not linearly, but grows nonetheless) by adding additional neutron API endpoints and RPC agents. OVN does not really horizontally
scale by adding additional API endpoints, from what we have observed.<br>
><br>
> 2. OVN gets significantly slower as load on the system grows. We have observed a soft cap of about 2000-2500 instances in a given deployment before ovn-backed neutron stops responding altogether to nova requests (even for booting a single instance). We have
observed linuxbridge get to 5000+ instances before it starts to struggle on the same hardware (and we think that linuxbridge can go further with improved provider network design in that particular case).<br>
><br>
> 3. Once the southbound database process hits 100% CPU usage on the leader in the ovn cluster, it’s game over (probably causes 1+2)<br>
><br>
><br>
><br>
> It's entirely possible that we just don’t understand OVN well enough to tune it [2][3][4], but then the question becomes how do we get that tuning knowledge into the docs so people don’t scratch their heads when their cool new OVN deployment scales 40% as
well as their ancient linuxbridge-based one?<br>
><br>
><br>
><br>
> If it is ‘known’ that OVN has some scaling challenges, is there a plan to fix it, and what is the best way to contribute to doing so?<br>
><br>
><br>
><br>
> We have observed similar results on Ubuntu 18.04/20.04 and CentOS 7/8 on Stein, Train, and Ussuri.<br>
><br>
><br>
><br>
> [1] <a href="https://pastebin.com/kyyURTJm" target="_blank">https://pastebin.com/kyyURTJm</a><br>
><br>
> [2] <a href="https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/ovsdb" target="_blank">https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/ovsdb</a><br>
><br>
> [3] <a href="https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/neutron" target="_blank">https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/neutron</a><br>
><br>
> [4] <a href="https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/compute" target="_blank">https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/compute</a><br>
><br>
><br>
><br>
> Chris Apsey<br>
><br>
> GEORGIA CYBER CENTER<br>
><br>
><u></u><u></u></p>
</div>
</div>
</blockquote></div></div>