<div dir="ltr"><p dir="ltr">Hi James, </p>
<p dir="ltr">This is awesome. I seem to have misplaced my 540-node cluster. ;-)</p><p>Is it possible for you to also patch in <a href="https://review.openstack.org/#/c/132372/">https://review.openstack.org/#/c/132372/</a> ? In my rally testing of port retrieval, this one probably made the most significant improvement.</p><p dir="ltr"><br></p>
<div class="gmail_quote">On Nov 12, 2014 9:26 AM, "James Page" <<a href="mailto:james.page@ubuntu.com" target="_blank">james.page@ubuntu.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA256<br>
<br>
Hi Ihar<br>
<br>
On 11/11/14 19:39, Ihar Hrachyshka wrote:<br>
>> there is a series of Neutron backports in the Juno queue that are<br>
>><br>
>>> intended to significantly improve service performance when<br>
>>> handling security groups (one of the issues that are main pain<br>
>>> points of current users): - <a href="https://review.openstack.org/130101" target="_blank">https://review.openstack.org/130101</a><br>
>>> - <a href="https://review.openstack.org/130098" target="_blank">https://review.openstack.org/130098</a> -<br>
>>> <a href="https://review.openstack.org/130100" target="_blank">https://review.openstack.org/130100</a> -<br>
>>> <a href="https://review.openstack.org/130097" target="_blank">https://review.openstack.org/130097</a> -<br>
>>> <a href="https://review.openstack.org/130105" target="_blank">https://review.openstack.org/130105</a> The first four patches are<br>
>>> optimizing db side (controller), while the last one is to avoid<br>
>>> fetching security group rules by OVS agent when firewall is<br>
>>> disabled.<br>
<br>
In terms of putting some figures around how these proposed stable<br>
patches help improve a Neutron based Juno cloud, I can provide some<br>
metrics based on recent testing that Canonical did in-conjuction with HP.<br>
<br>
The cloud we deployed was all based on Intel Atom Quad Core processors,<br>
with 16G of RAM and SSD disk; 540 servers in total including 8 nova<br>
controllers and 4 neutron controllers. OpenStack Juno release on<br>
Ubuntu 14.04.<br>
<br>
With around 12,000 running instances, which was as far as I could push<br>
a vanilla ML2/ovs based Juno cloud, the load on the 4 neutron<br>
controllers was around 40 with CPU maxing out all of the time - which<br>
pretty much mean't it was impossible to create any new instances due<br>
to vif plugging timeouts in nova waiting for neutron to complete<br>
network setup.<br>
<br>
I patched in:<br>
<br>
<a href="https://review.openstack.org/#/c/130101/" target="_blank">https://review.openstack.org/#/c/130101/</a><br>
<a href="https://review.openstack.org/#/c/130098/" target="_blank">https://review.openstack.org/#/c/130098/</a><br>
<a href="https://review.openstack.org/#/c/130100/" target="_blank">https://review.openstack.org/#/c/130100/</a><br>
<a href="https://review.openstack.org/#/c/130105/" target="_blank">https://review.openstack.org/#/c/130105/</a><br>
<br>
and re-ran the same test; the messaging load on the RabbitMQ server at<br>
12,000 instances was considerably less in terms of volume, and the<br>
load on the 4 neutron controllers was around 10 (vs 40) with CPU at<br>
around 55->65% utilization - so still pretty busy, but a better<br>
situation than without the patches.<br>
<br>
My testing was quite synthetic (boot small instances until things<br>
start to break) but it does illustrate the difference these patches make.<br>
<br>
HTH<br>
<br>
James<br>
<br>
- --<br>
James Page<br>
Ubuntu and Debian Developer<br>
<a href="mailto:james.page@ubuntu.com" target="_blank">james.page@ubuntu.com</a><br>
<a href="mailto:jamespage@debian.org" target="_blank">jamespage@debian.org</a><br>
-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v1<br>
<br>
iQIcBAEBCAAGBQJUY5czAAoJEL/srsug59jDIJ8QAJAK8aWUSQGyPrcAqvKi+OE7<br>
vS4NhVWog1ifubPcIpDstAoELHIfQVQKaryCN7oXAzQ0Yyp68DE68mw+o8rrly2S<br>
25gPebORNath1BOJMMlv5iRS0lVN30cfmRrs9nfQ5bdAE6qkaPlofG9GsGRggCG2<br>
feewRR9w+PFFQQ9NdsZ141FoQDtpLjhY095rEwzUhyah8spM2w2er2XiEJLHRTI/<br>
HcJybUSX/Nu8OV4FJ6dn+pebWv1iWgzNOV/eqCYHf1Mx9G6HrB8ZQpv486LznyX1<br>
PSNuiVMgUFcSWUcN1lFQSEe/ASW+G2t3/aEMKZBXiXsO3DTORtZ79oCTkzipkehj<br>
18ztLr+nkCDrdGzbvkD6LWGt9F7MjTzsXao4RwGe/EiRBvcrvnHpkc5kfaW2aIb3<br>
+rH8pcHpfaC04y7Zy492lFrkmrXn+73c2a+hS+gS3bMmQ1bcwF+QeeXunsMajgVo<br>
CQW98n3HJI/jAjCBEbV5cmmw+BXQDWOHYlP+tZiAMC5Tnj42/9+K+KWZr+truhLK<br>
cKGFlM+vaVsykAh9KIf1E/e6G72o/kihXDUnpx/mSk27sxDILEz9ItcQRJgpQCPN<br>
cH3sIj+qG76NDqIhdLYs8LgyjwQI2SdOeSi+32oCCe3tnaI35FKKuRMI0oSP0HKn<br>
3U7bekTsjXhlBWusW9Wb<br>
=WaFI<br>
-----END PGP SIGNATURE-----<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div>
</div>