<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Yea - I think those patches would help a lot. Especially the security group related change. Security groups for some reason are the most expensive call in Neutron for us. In our larger deployments the simplest security group list commands takes 60 seconds to
perform. We had very similar issues with neutron-lbaas, but those calls has since been fixed.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The large ops SIG is unfortunately 1AM (Pacific Time) over here. I can try to attend it, but wouldn't be easy.</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Matt Riedemann <mriedemos@gmail.com><br>
<b>Sent:</b> Tuesday, December 3, 2019 9:44 AM<br>
<b>To:</b> openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org><br>
<b>Subject:</b> Re: [neutron] api performance at scale</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">On 12/3/2019 11:24 AM, Erik Olof Gunnar Andersson wrote:<br>
> For us nova used to be the biggest concern, but a lot of work has been <br>
> done and nova now performers great. Instead we are having issues to get <br>
> Neutron to perform at scale. Obvious calls like security groups are <br>
> performing really poorly, and nova-compute defaults for refreshing the <br>
> network cache on computes causes massive issues with Neutron.<br>
<br>
I wonder how much of the performance hit is due to rootwrap usage in <br>
neutron (nova's conversion to privsep was completed in Train).<br>
<br>
Nova might be the bees knees, but I know there are things in nova we <br>
could do to be smarter about not hammering the neutron API as much, e.g.:<br>
<br>
<a href="https://urldefense.com/v3/__https://review.opendev.org/*/c/465792/__;Iw!2E0gRdhhnqPNNL0!0A97ZwnFJg3RpxtEwi5sVytDBtU_R8YdJ5P9h8OAUX7ciGEtHKVExLKEUIssCQ7svA$">https://urldefense.com/v3/__https://review.opendev.org/*/c/465792/__;Iw!2E0gRdhhnqPNNL0!0A97ZwnFJg3RpxtEwi5sVytDBtU_R8YdJ5P9h8OAUX7ciGEtHKVExLKEUIssCQ7svA$</a>
- make bulk queries to neutron <br>
when refreshing the instance network info cache<br>
<br>
<a href="https://urldefense.com/v3/__https://review.opendev.org/*/q/I7de14456d04370c842b4c35597dca3a628a826a2__;Iw!2E0gRdhhnqPNNL0!0A97ZwnFJg3RpxtEwi5sVytDBtU_R8YdJ5P9h8OAUX7ciGEtHKVExLKEUIsamDve8Q$">https://urldefense.com/v3/__https://review.opendev.org/*/q/I7de14456d04370c842b4c35597dca3a628a826a2__;Iw!2E0gRdhhnqPNNL0!0A97ZwnFJg3RpxtEwi5sVytDBtU_R8YdJ5P9h8OAUX7ciGEtHKVExLKEUIsamDve8Q$</a>
<br>
- be smarter about filtering to avoid expensive joins<br>
<br>
<a href="https://urldefense.com/v3/__https://bugs.launchpad.net/nova/*bug/1567655__;Kw!2E0gRdhhnqPNNL0!0A97ZwnFJg3RpxtEwi5sVytDBtU_R8YdJ5P9h8OAUX7ciGEtHKVExLKEUIubIZxDAg$">https://urldefense.com/v3/__https://bugs.launchpad.net/nova/*bug/1567655__;Kw!2E0gRdhhnqPNNL0!0A97ZwnFJg3RpxtEwi5sVytDBtU_R8YdJ5P9h8OAUX7ciGEtHKVExLKEUIubIZxDAg$</a>
- nova's internal network <br>
info cache only stores information about ports and their related <br>
networks/subnets/ips but the security group information related to the <br>
ports attached to a server is fetched directly anytime it's needed, <br>
including when listing servers with details. So if you're an admin <br>
listing all servers across all tenants, that could get pretty slow. I've <br>
long thought we should cache the security group information like we do <br>
for ports for read-only operations like GET /servers/detail but it's a <br>
non-trivial amount of work to make that happen and we'd definitely want <br>
benchmarks and such to justify the change.<br>
<br>
Note ttx has started a large ops SIG or whatever so this is probably <br>
something to discuss there:<br>
<br>
<a href="https://urldefense.com/v3/__https://wiki.openstack.org/wiki/Large_Scale_SIG__;!2E0gRdhhnqPNNL0!0A97ZwnFJg3RpxtEwi5sVytDBtU_R8YdJ5P9h8OAUX7ciGEtHKVExLKEUItHA86njA$">https://urldefense.com/v3/__https://wiki.openstack.org/wiki/Large_Scale_SIG__;!2E0gRdhhnqPNNL0!0A97ZwnFJg3RpxtEwi5sVytDBtU_R8YdJ5P9h8OAUX7ciGEtHKVExLKEUItHA86njA$</a>
<br>
<br>
-- <br>
<br>
Thanks,<br>
<br>
Matt<br>
<br>
</div>
</span></font></div>
</body>
</html>