I'd like to see something like this automated as part of the smoke tests, otherwise its easy for performance regressions to sneak in. I am not aware of this existing for a Folsom-based setup yet. <div><br></div><div>
This seems to be the bug we're using to track: <a href="https://bugs.launchpad.net/quantum/+bug/1075369">https://bugs.launchpad.net/quantum/+bug/1075369</a>. I don't see a linked review on gerrit though. Have you not yet posted it? </div>
<div><br></div><div>Dan</div><div><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Nov 7, 2012 at 7:56 AM, Debojyoti Dutta <span dir="ltr"><<a href="mailto:ddutta@gmail.com" target="_blank">ddutta@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><br></div><div>In fact this points to the fact that we need a Geekbench equivalent for openstack so that folks can submit performance numbers for their scaled setups. </div>
<div><br></div><div>For starters, we could have simple metrics like the test here (for quantum) and repeat this for nova too (for starting a bunch of VMs). </div>
<div><br></div><div>The submitted numbers will also be a metric for new fixes/features which might have performance sideeffects</div><div><br></div><div>debo</div><div><br></div><div class="gmail_extra"><div><div class="h5">
<br><br><div class="gmail_quote">
On Wed, Nov 7, 2012 at 7:20 AM, Edgar Magana (eperdomo) <span dir="ltr"><<a href="mailto:eperdomo@cisco.com" target="_blank">eperdomo@cisco.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="font-size:14px;font-family:Calibri,sans-serif;word-wrap:break-word">
<div>Team,</div>
<div><br>
</div>
<div>We ran a comparative tests between current Quantum code and our proposed fix for this bug and the results are incredible, in current quantum code it takes around 4-5 seconds to create a network when already have over 600 networks created and almost 7 second
when we have over 1K networks. See this log:</div>
<div><br>
</div>
<div>
<div>Tue Nov 6 23:39:52 PST 2012</div>
<div>Created a new network:</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| admin_state_up | True |</div>
<div>| id | e87b2f27-9308-4001-b180-228ff5ab6479 |</div>
<div>| name | net621 |</div>
<div>| provider:network_type | local |</div>
<div>| provider:physical_network | |</div>
<div>| provider:segmentation_id | |</div>
<div>| router:external | False |</div>
<div>| shared | False |</div>
<div>| status | ACTIVE |</div>
<div>| subnets | |</div>
<div>| tenant_id | b248e18b173b4ea99873e102a0715dcd |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>Tue Nov 6 23:39:57 PST 2012</div>
</div>
<div>++++</div>
<div>
<div>Wed Nov 7 00:17:46 PST 2012</div>
<div>Created a new network:</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| admin_state_up | True |</div>
<div>| id | 2138b910-8996-40b7-a9b4-ac090e126a2b |</div>
<div>| name | net1032 |</div>
<div>| provider:network_type | local |</div>
<div>| provider:physical_network | |</div>
<div>| provider:segmentation_id | |</div>
<div>| router:external | False |</div>
<div>| shared | False |</div>
<div>| status | ACTIVE |</div>
<div>| subnets | |</div>
<div>| tenant_id | b248e18b173b4ea99873e102a0715dcd |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>Wed Nov 7 00:17:53 PST 2012</div>
<div>Wed Nov 7 00:17:53 PST 2012</div>
<div>Created a new network:</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| admin_state_up | True |</div>
<div>| id | 29530fb3-1a1a-4c88-b9e4-954988bb8ccb |</div>
<div>| name | net1033 |</div>
<div>| provider:network_type | local |</div>
<div>| provider:physical_network | |</div>
<div>| provider:segmentation_id | |</div>
<div>| router:external | False |</div>
<div>| shared | False |</div>
<div>| status | ACTIVE |</div>
<div>| subnets | |</div>
<div>| tenant_id | b248e18b173b4ea99873e102a0715dcd |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>Wed Nov 7 00:18:00 PST 2012</div>
</div>
<div><br>
</div>
<div>With the changes on the count call we have a constant performance regardless the number of networks created:</div>
<div><br>
</div>
<div>
<div>Wed Nov 7 02:48:59 PST 2012</div>
<div>Created a new network:</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| admin_state_up | True |</div>
<div>| id | 024aa164-cb7b-4683-a5ae-41f11a582529 |</div>
<div>| name | net3847 |</div>
<div>| provider:network_type | local |</div>
<div>| provider:physical_network | |</div>
<div>| provider:segmentation_id | |</div>
<div>| router:external | False |</div>
<div>| shared | False |</div>
<div>| status | ACTIVE |</div>
<div>| subnets | |</div>
<div>| tenant_id | 5baba10045db44cdad235fff1d5e59b1 |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>Wed Nov 7 02:49:00 PST 2012</div>
<div>Wed Nov 7 02:49:00 PST 2012</div>
<div>Created a new network:</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| admin_state_up | True |</div>
<div>| id | 91e7eaa3-bdc4-4705-a8b5-2b63500fd066 |</div>
<div>| name | net3848 |</div>
<div>| provider:network_type | local |</div>
<div>| provider:physical_network | |</div>
<div>| provider:segmentation_id | |</div>
<div>| router:external | False |</div>
<div>| shared | False |</div>
<div>| status | ACTIVE |</div>
<div>| subnets | |</div>
<div>| tenant_id | 5baba10045db44cdad235fff1d5e59b1 |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>Wed Nov 7 02:49:01 PST 2012</div>
<div>Wed Nov 7 02:49:01 PST 2012</div>
<div>Created a new network:</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| admin_state_up | True |</div>
<div>| id | 19fd1785-77c2-4526-9fa6-5a2c8c01b3a4 |</div>
<div>| name | net3849 |</div>
<div>| provider:network_type | local |</div>
<div>| provider:physical_network | |</div>
<div>| provider:segmentation_id | |</div>
<div>| router:external | False |</div>
<div>| shared | False |</div>
<div>| status | ACTIVE |</div>
<div>| subnets | |</div>
<div>| tenant_id | 5baba10045db44cdad235fff1d5e59b1 |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>Wed Nov 7 02:49:01 PST 2012</div>
</div>
<div><br>
</div>
<div>We will push this code changes ASAP.</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Edgar</div>
<div><br>
</div>
<span>
<div style="border-right:medium none;padding-right:0in;padding-left:0in;padding-top:3pt;text-align:left;font-size:11pt;border-bottom:medium none;font-family:Calibri;border-top:#b5c4df 1pt solid;padding-bottom:0in;border-left:medium none">
<span style="font-weight:bold">From: </span>Edgar Magana <<a href="mailto:eperdomo@cisco.com" target="_blank">eperdomo@cisco.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>OpenStack List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Tuesday, November 6, 2012 9:19 AM<br>
<span style="font-weight:bold">To: </span>OpenStack List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>>, "<a href="mailto:gkotton@redhat.com" target="_blank">gkotton@redhat.com</a>" <<a href="mailto:gkotton@redhat.com" target="_blank">gkotton@redhat.com</a>><div>
<div><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] Quantum Performance<br>
</div></div></div><div><div>
<div><br>
</div>
<div>
<div style="word-wrap:break-word">
<div style="font-size:14px;font-family:Calibri,sans-serif">
Hi Folks,</div>
<div style="font-size:14px;font-family:Calibri,sans-serif">
<br>
</div>
<div style="font-size:14px;font-family:Calibri,sans-serif">
I just want to give you an update on this topic, we opened a bug against this behavior:</div>
<div style="font-size:14px;font-family:Calibri,sans-serif">
<a href="https://bugs.launchpad.net/quantum/+bug/1075369" target="_blank">https://bugs.launchpad.net/quantum/+bug/1075369</a></div>
<div style="font-size:14px;font-family:Calibri,sans-serif">
<br>
</div>
<div><span style="line-height:18px;text-align:left">Bug Description:</span></div>
<div><span style="line-height:18px;text-align:left">In quantum when a new network is created and system checks for user's quota, instead of getting a count from DB object, it returns all objects from db and locally
counts them all. It has performance implications when the number of objects increases.</span></div>
<div><span style="line-height:18px;text-align:left"><br>
</span></div>
<div><span style="line-height:18px;text-align:left;background-color:rgb(255,254,254)">We will submit the patch ASAP.</span></div>
<div><span style="line-height:18px;text-align:left;background-color:rgb(255,254,254)"><br>
</span></div>
<div><span style="line-height:18px;text-align:left;background-color:rgb(254,253,253)">Thanks,</span></div>
<div><span style="line-height:18px;text-align:left;background-color:rgb(254,253,253)"><br>
</span></div>
<div><span style="line-height:18px;text-align:left;background-color:rgb(253,252,252)">Edgar</span></div>
<div style="font-size:14px;font-family:Calibri,sans-serif">
<br>
</div>
<span style="font-size:14px;font-family:Calibri,sans-serif">
<div style="border-right:medium none;padding-right:0in;padding-left:0in;padding-top:3pt;text-align:left;font-size:11pt;border-bottom:medium none;font-family:Calibri;border-top:#b5c4df 1pt solid;padding-bottom:0in;border-left:medium none">
<span style="font-weight:bold">From: </span>Salvatore Orlando <<a href="mailto:sorlando@nicira.com" target="_blank">sorlando@nicira.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>OpenStack List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, October 31, 2012 12:54 AM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:gkotton@redhat.com" target="_blank">gkotton@redhat.com</a>" <<a href="mailto:gkotton@redhat.com" target="_blank">gkotton@redhat.com</a>>, OpenStack List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] Quantum Performance<br>
</div>
<div><br>
</div>
<div>
<div>Edgar,
<div><br>
</div>
<div>My suspicion here goes to the IP allocation mechanisms.</div>
<div>Hence, I'd try to run the following tests (they should be easily scriptable):</div>
<div><br>
</div>
<div>1) Create networks no subnets</div>
<div>2) Create networks, with subnet having a small CIDR (say /28)</div>
<div>3) Create networks with larger CIDRs (say /24, /20, /16 etc)</div>
<div><br>
</div>
<div>What behaviour do you observe in the above three cases?</div>
<div><br>
</div>
<div>Also, It might be worth shutting down agent to pinpoint whether the problem is in Quantum service itself on in the interactions with the various agents.</div>
<div><br>
</div>
<div>Salvatore</div>
<div>
<div><br>
</div>
<div>PS: thanks for doing these tests - They've been lying on my TODO list for way too much time!</div>
<div>If you don't hate me too much for the service insertion story, I'll be glad to buy you a drink at the next summit :)</div>
<div><br>
</div>
<br>
<div class="gmail_quote">On 31 October 2012 08:37, Gary Kotton <span dir="ltr"><<a href="mailto:gkotton@redhat.com" target="_blank">gkotton@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>On 10/31/2012 09:48 AM, Edgar Magana (eperdomo) wrote:
<blockquote type="cite">
<div>Hi Folks,</div>
<div><br>
</div>
<div>I am running OpenStack Folsom with Quantum/OVS plugin. I am running some performance tests for Quantum, basically I am creating 5000 network with one subnet each one of them.</div>
<div>After the first ~50 – 70 networks the system response time is slower and slower, to the point that it could take up to ~6 - 7 seconds to create just a network and the same for the subnets.</div>
</blockquote>
<br>
</div>
Hi,<br>
Is the problem you are seeing with the Quantum service or the agents? Off the bat I would say that it is with the Quantum service. Are the calls done in parallel or are they done sequentially?<br>
<br>
There are a number of things that we should do to isolate this:<br>
1. Check the internal logic of the network creation. Basically there are 3 stages:<br>
i. provider network treatment<br>
ii network create<br>
iii. l3 notification<br>
It would be interesting to see who eats the most time.<br>
2. Need to profile database access<br>
<br>
Thanks<br>
Gary<br>
<br>
<blockquote type="cite">
<div>
<div>I would like to know if somebody else has experimented this kind of behavior in Quantum or even in nova-network.</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Edgar</div>
<div> </div>
<br>
<fieldset></fieldset> <br>
</div>
<pre>_______________________________________________
OpenStack-dev mailing list
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></pre>
</blockquote>
<br>
</div>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</span></div>
</div>
</div></div></span>
</div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br>-Debo~<br><br>
</font></span></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>Dan Wendlandt <div>Nicira, Inc: <a href="http://www.nicira.com" target="_blank">www.nicira.com</a><br><div>twitter: danwendlandt<br>
~~~~~~~~~~~~~~~~~~~~~~~~~~~<br></div></div><br>
</div>