<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Helvetica LT Std Black";}
@font-face
{font-family:"Helvetica LT Std Cond Light";}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Assaf,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">We can absolutely support engineering poking around in our environment (and possibly an even larger one at my previous employer that was experiencing similar
issues during testing). We can take this offline so we don’t spam the mailing list.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Just let me know how to proceed,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Helvetica LT Std Cond Light";color:black">Chris Apsey<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Helvetica LT Std Black";color:#C1D72E">GEORGIA CYBER CENTER<o:p></o:p></span></b></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Assaf Muller <amuller@redhat.com>
<br>
<b>Sent:</b> Thursday, August 27, 2020 11:18 AM<br>
<b>To:</b> Apsey, Christopher <CAPSEY@augusta.edu><br>
<b>Cc:</b> openstack-discuss@lists.openstack.org; Lucas Alvares Gomes Martins <lmartins@redhat.com>; Jakub Libosvar <jlibosva@redhat.com>; Daniel Alvarez Sanchez <dalvarez@redhat.com><br>
<b>Subject:</b> [EXTERNAL] Re: [neutron][ovn] OVN Performance<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">CAUTION: EXTERNAL SENDER This email originated from an external source. Please exercise caution before opening attachments, clicking links, replying, or providing information to the sender. If you believe it to be fraudulent, contact the
AU Cybersecurity Hotline at 72-CYBER (2-9237 / 706-722-9237) or <a href="mailto:72CYBER@augusta.edu">
72CYBER@augusta.edu</a><br>
<br>
The most efficient way about this is to give one or more of the<br>
Engineers working on OpenStack OVN upstream (I've added a few to this<br>
thread) temporary access to an environment that can reproduce issues<br>
you're seeing, we could then document the issues and work towards<br>
solutions. If that's not possible, if you could provide reproducer<br>
scripts, or alternatively sharpen the reproduction method, we'll take<br>
a look. What you've described is not something that's 'acceptable',<br>
OVN should definitely not scale worse than Neutron with the Linux<br>
Bridge agent. It's possible that the particular issues you ran in to<br>
is something that we've already seen internally at Red Hat, or with<br>
our customers, and we're already working on fixes in future versions<br>
of OVN - I can't tell you until you elaborate on the details of the<br>
issues you're seeing. In any case, the upstream community is committed<br>
to improving OVN scale and fixing scale issues as they pop up.<br>
Coincidentally, Red Hat scale engineers just published an article [1]<br>
about work they've done to scale RH-OSP 16.1 (== OpenStack Train on<br>
CentOS 8, with OVN 2.13 and TripleO) to 700 compute nodes.<br>
<br>
[1] <a href="https://www.redhat.com/en/blog/scaling-red-hat-openstack-platform-161-more-700-nodes?source=bloglisting">https://www.redhat.com/en/blog/scaling-red-hat-openstack-platform-161-more-700-nodes?source=bloglisting</a><br>
<br>
On Thu, Aug 27, 2020 at 10:44 AM Apsey, Christopher <<a href="mailto:CAPSEY@augusta.edu">CAPSEY@augusta.edu</a>> wrote:<br>
><br>
> All,<br>
><br>
><br>
><br>
> I know that OVN is going to become the default neutron backend at some point and displace linuxbridge as the default configuration option in the docs, but we have noticed a pretty significant performance disparity between OVN and linuxbridge on identical
hardware over the past year or so in a few different environments[1]. I know that example is unscientific, but similar results have been borne out in many different scenarios from what we have observed. There are three main problems from what we see:<br>
><br>
><br>
><br>
> 1. OVN does not handle large concurrent requests as well as linuxbridge. Additionally, linuxbridge concurrent capacity grows (not linearly, but grows nonetheless) by adding additional neutron API endpoints and RPC agents. OVN does not really horizontally
scale by adding additional API endpoints, from what we have observed.<br>
><br>
> 2. OVN gets significantly slower as load on the system grows. We have observed a soft cap of about 2000-2500 instances in a given deployment before ovn-backed neutron stops responding altogether to nova requests (even for booting a single instance). We have
observed linuxbridge get to 5000+ instances before it starts to struggle on the same hardware (and we think that linuxbridge can go further with improved provider network design in that particular case).<br>
><br>
> 3. Once the southbound database process hits 100% CPU usage on the leader in the ovn cluster, it’s game over (probably causes 1+2)<br>
><br>
><br>
><br>
> It's entirely possible that we just don’t understand OVN well enough to tune it [2][3][4], but then the question becomes how do we get that tuning knowledge into the docs so people don’t scratch their heads when their cool new OVN deployment scales 40% as
well as their ancient linuxbridge-based one?<br>
><br>
><br>
><br>
> If it is ‘known’ that OVN has some scaling challenges, is there a plan to fix it, and what is the best way to contribute to doing so?<br>
><br>
><br>
><br>
> We have observed similar results on Ubuntu 18.04/20.04 and CentOS 7/8 on Stein, Train, and Ussuri.<br>
><br>
><br>
><br>
> [1] <a href="https://pastebin.com/kyyURTJm">https://pastebin.com/kyyURTJm</a><br>
><br>
> [2] <a href="https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/ovsdb">https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/ovsdb</a><br>
><br>
> [3] <a href="https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/neutron">https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/neutron</a><br>
><br>
> [4] <a href="https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/compute">https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/compute</a><br>
><br>
><br>
><br>
> Chris Apsey<br>
><br>
> GEORGIA CYBER CENTER<br>
><br>
><o:p></o:p></p>
</div>
</body>
</html>