<br><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Nov 7, 2012 at 10:12 AM, Dan Wendlandt <span dir="ltr"><<a href="mailto:dan@nicira.com" target="_blank">dan@nicira.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'd like to see something like this automated as part of the smoke tests, otherwise its easy for performance regressions to sneak in. I am not aware of this existing for a Folsom-based setup yet. <div><br></div><div>
This seems to be the bug we're using to track: <a href="https://bugs.launchpad.net/quantum/+bug/1075369" target="_blank">https://bugs.launchpad.net/quantum/+bug/1075369</a>. I don't see a linked review on gerrit though. Have you not yet posted it? </div>
</blockquote><div><br></div><div>Actually, looking at the code, the change is pretty simple. The only real trick is to handle the fact that a true "count_x" resource is not part of the plugin API, but would make sense given quotas. Since we can't change the plugin API after the fact for Folsom, it probably makes sense to have the quota code check if a count_x method exists, and if not, fall back to using the existing method. For any plugin that inherits from db_base_plugin_v2, we can provide default count_x methods that use sqlalchemy.</div>
<div><br></div><div>Dan</div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="HOEnZb"><font color="#888888">
<div><br></div><div>Dan</div><div><br></div></font></span><div class="gmail_extra"><div><div class="h5"><br><br><div class="gmail_quote">On Wed, Nov 7, 2012 at 7:56 AM, Debojyoti Dutta <span dir="ltr"><<a href="mailto:ddutta@gmail.com" target="_blank">ddutta@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><br></div><div>In fact this points to the fact that we need a Geekbench equivalent for openstack so that folks can submit performance numbers for their scaled setups. </div>
<div><br></div><div>For starters, we could have simple metrics like the test here (for quantum) and repeat this for nova too (for starting a bunch of VMs). </div>
<div><br></div><div>The submitted numbers will also be a metric for new fixes/features which might have performance sideeffects</div><div><br></div><div>debo</div><div><br></div><div class="gmail_extra"><div><div>
<br><br><div class="gmail_quote">
On Wed, Nov 7, 2012 at 7:20 AM, Edgar Magana (eperdomo) <span dir="ltr"><<a href="mailto:eperdomo@cisco.com" target="_blank">eperdomo@cisco.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="font-size:14px;font-family:Calibri,sans-serif;word-wrap:break-word">
<div>Team,</div>
<div><br>
</div>
<div>We ran a comparative tests between current Quantum code and our proposed fix for this bug and the results are incredible, in current quantum code it takes around 4-5 seconds to create a network when already have over 600 networks created and almost 7 second
when we have over 1K networks. See this log:</div>
<div><br>
</div>
<div>
<div>Tue Nov 6 23:39:52 PST 2012</div>
<div>Created a new network:</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| admin_state_up | True |</div>
<div>| id | e87b2f27-9308-4001-b180-228ff5ab6479 |</div>
<div>| name | net621 |</div>
<div>| provider:network_type | local |</div>
<div>| provider:physical_network | |</div>
<div>| provider:segmentation_id | |</div>
<div>| router:external | False |</div>
<div>| shared | False |</div>
<div>| status | ACTIVE |</div>
<div>| subnets | |</div>
<div>| tenant_id | b248e18b173b4ea99873e102a0715dcd |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>Tue Nov 6 23:39:57 PST 2012</div>
</div>
<div>++++</div>
<div>
<div>Wed Nov 7 00:17:46 PST 2012</div>
<div>Created a new network:</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| admin_state_up | True |</div>
<div>| id | 2138b910-8996-40b7-a9b4-ac090e126a2b |</div>
<div>| name | net1032 |</div>
<div>| provider:network_type | local |</div>
<div>| provider:physical_network | |</div>
<div>| provider:segmentation_id | |</div>
<div>| router:external | False |</div>
<div>| shared | False |</div>
<div>| status | ACTIVE |</div>
<div>| subnets | |</div>
<div>| tenant_id | b248e18b173b4ea99873e102a0715dcd |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>Wed Nov 7 00:17:53 PST 2012</div>
<div>Wed Nov 7 00:17:53 PST 2012</div>
<div>Created a new network:</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| admin_state_up | True |</div>
<div>| id | 29530fb3-1a1a-4c88-b9e4-954988bb8ccb |</div>
<div>| name | net1033 |</div>
<div>| provider:network_type | local |</div>
<div>| provider:physical_network | |</div>
<div>| provider:segmentation_id | |</div>
<div>| router:external | False |</div>
<div>| shared | False |</div>
<div>| status | ACTIVE |</div>
<div>| subnets | |</div>
<div>| tenant_id | b248e18b173b4ea99873e102a0715dcd |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>Wed Nov 7 00:18:00 PST 2012</div>
</div>
<div><br>
</div>
<div>With the changes on the count call we have a constant performance regardless the number of networks created:</div>
<div><br>
</div>
<div>
<div>Wed Nov 7 02:48:59 PST 2012</div>
<div>Created a new network:</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| admin_state_up | True |</div>
<div>| id | 024aa164-cb7b-4683-a5ae-41f11a582529 |</div>
<div>| name | net3847 |</div>
<div>| provider:network_type | local |</div>
<div>| provider:physical_network | |</div>
<div>| provider:segmentation_id | |</div>
<div>| router:external | False |</div>
<div>| shared | False |</div>
<div>| status | ACTIVE |</div>
<div>| subnets | |</div>
<div>| tenant_id | 5baba10045db44cdad235fff1d5e59b1 |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>Wed Nov 7 02:49:00 PST 2012</div>
<div>Wed Nov 7 02:49:00 PST 2012</div>
<div>Created a new network:</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| admin_state_up | True |</div>
<div>| id | 91e7eaa3-bdc4-4705-a8b5-2b63500fd066 |</div>
<div>| name | net3848 |</div>
<div>| provider:network_type | local |</div>
<div>| provider:physical_network | |</div>
<div>| provider:segmentation_id | |</div>
<div>| router:external | False |</div>
<div>| shared | False |</div>
<div>| status | ACTIVE |</div>
<div>| subnets | |</div>
<div>| tenant_id | 5baba10045db44cdad235fff1d5e59b1 |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>Wed Nov 7 02:49:01 PST 2012</div>
<div>Wed Nov 7 02:49:01 PST 2012</div>
<div>Created a new network:</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| admin_state_up | True |</div>
<div>| id | 19fd1785-77c2-4526-9fa6-5a2c8c01b3a4 |</div>
<div>| name | net3849 |</div>
<div>| provider:network_type | local |</div>
<div>| provider:physical_network | |</div>
<div>| provider:segmentation_id | |</div>
<div>| router:external | False |</div>
<div>| shared | False |</div>
<div>| status | ACTIVE |</div>
<div>| subnets | |</div>
<div>| tenant_id | 5baba10045db44cdad235fff1d5e59b1 |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>Wed Nov 7 02:49:01 PST 2012</div>
</div>
<div><br>
</div>
<div>We will push this code changes ASAP.</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Edgar</div>
<div><br>
</div>
<span>
<div style="border-right:medium none;padding-right:0in;padding-left:0in;padding-top:3pt;text-align:left;font-size:11pt;border-bottom:medium none;font-family:Calibri;border-top:#b5c4df 1pt solid;padding-bottom:0in;border-left:medium none">
<span style="font-weight:bold">From: </span>Edgar Magana <<a href="mailto:eperdomo@cisco.com" target="_blank">eperdomo@cisco.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>OpenStack List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Tuesday, November 6, 2012 9:19 AM<br>
<span style="font-weight:bold">To: </span>OpenStack List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>>, "<a href="mailto:gkotton@redhat.com" target="_blank">gkotton@redhat.com</a>" <<a href="mailto:gkotton@redhat.com" target="_blank">gkotton@redhat.com</a>><div>
<div><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] Quantum Performance<br>
</div></div></div><div><div>
<div><br>
</div>
<div>
<div style="word-wrap:break-word">
<div style="font-size:14px;font-family:Calibri,sans-serif">
Hi Folks,</div>
<div style="font-size:14px;font-family:Calibri,sans-serif">
<br>
</div>
<div style="font-size:14px;font-family:Calibri,sans-serif">
I just want to give you an update on this topic, we opened a bug against this behavior:</div>
<div style="font-size:14px;font-family:Calibri,sans-serif">
<a href="https://bugs.launchpad.net/quantum/+bug/1075369" target="_blank">https://bugs.launchpad.net/quantum/+bug/1075369</a></div>
<div style="font-size:14px;font-family:Calibri,sans-serif">
<br>
</div>
<div><span style="line-height:18px;text-align:left">Bug Description:</span></div>
<div><span style="line-height:18px;text-align:left">In quantum when a new network is created and system checks for user's quota, instead of getting a count from DB object, it returns all objects from db and locally
counts them all. It has performance implications when the number of objects increases.</span></div>
<div><span style="line-height:18px;text-align:left"><br>
</span></div>
<div><span style="line-height:18px;text-align:left;background-color:rgb(255,254,254)">We will submit the patch ASAP.</span></div>
<div><span style="line-height:18px;text-align:left;background-color:rgb(255,254,254)"><br>
</span></div>
<div><span style="line-height:18px;text-align:left;background-color:rgb(254,253,253)">Thanks,</span></div>
<div><span style="line-height:18px;text-align:left;background-color:rgb(254,253,253)"><br>
</span></div>
<div><span style="line-height:18px;text-align:left;background-color:rgb(253,252,252)">Edgar</span></div>
<div style="font-size:14px;font-family:Calibri,sans-serif">
<br>
</div>
<span style="font-size:14px;font-family:Calibri,sans-serif">
<div style="border-right:medium none;padding-right:0in;padding-left:0in;padding-top:3pt;text-align:left;font-size:11pt;border-bottom:medium none;font-family:Calibri;border-top:#b5c4df 1pt solid;padding-bottom:0in;border-left:medium none">
<span style="font-weight:bold">From: </span>Salvatore Orlando <<a href="mailto:sorlando@nicira.com" target="_blank">sorlando@nicira.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>OpenStack List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, October 31, 2012 12:54 AM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:gkotton@redhat.com" target="_blank">gkotton@redhat.com</a>" <<a href="mailto:gkotton@redhat.com" target="_blank">gkotton@redhat.com</a>>, OpenStack List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] Quantum Performance<br>
</div>
<div><br>
</div>
<div>
<div>Edgar,
<div><br>
</div>
<div>My suspicion here goes to the IP allocation mechanisms.</div>
<div>Hence, I'd try to run the following tests (they should be easily scriptable):</div>
<div><br>
</div>
<div>1) Create networks no subnets</div>
<div>2) Create networks, with subnet having a small CIDR (say /28)</div>
<div>3) Create networks with larger CIDRs (say /24, /20, /16 etc)</div>
<div><br>
</div>
<div>What behaviour do you observe in the above three cases?</div>
<div><br>
</div>
<div>Also, It might be worth shutting down agent to pinpoint whether the problem is in Quantum service itself on in the interactions with the various agents.</div>
<div><br>
</div>
<div>Salvatore</div>
<div>
<div><br>
</div>
<div>PS: thanks for doing these tests - They've been lying on my TODO list for way too much time!</div>
<div>If you don't hate me too much for the service insertion story, I'll be glad to buy you a drink at the next summit :)</div>
<div><br>
</div>
<br>
<div class="gmail_quote">On 31 October 2012 08:37, Gary Kotton <span dir="ltr"><<a href="mailto:gkotton@redhat.com" target="_blank">gkotton@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>On 10/31/2012 09:48 AM, Edgar Magana (eperdomo) wrote:
<blockquote type="cite">
<div>Hi Folks,</div>
<div><br>
</div>
<div>I am running OpenStack Folsom with Quantum/OVS plugin. I am running some performance tests for Quantum, basically I am creating 5000 network with one subnet each one of them.</div>
<div>After the first ~50 – 70 networks the system response time is slower and slower, to the point that it could take up to ~6 - 7 seconds to create just a network and the same for the subnets.</div>
</blockquote>
<br>
</div>
Hi,<br>
Is the problem you are seeing with the Quantum service or the agents? Off the bat I would say that it is with the Quantum service. Are the calls done in parallel or are they done sequentially?<br>
<br>
There are a number of things that we should do to isolate this:<br>
1. Check the internal logic of the network creation. Basically there are 3 stages:<br>
i. provider network treatment<br>
ii network create<br>
iii. l3 notification<br>
It would be interesting to see who eats the most time.<br>
2. Need to profile database access<br>
<br>
Thanks<br>
Gary<br>
<br>
<blockquote type="cite">
<div>
<div>I would like to know if somebody else has experimented this kind of behavior in Quantum or even in nova-network.</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Edgar</div>
<div> </div>
<br>
<fieldset></fieldset> <br>
</div>
<pre>_______________________________________________
OpenStack-dev mailing list
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></pre>
</blockquote>
<br>
</div>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</span></div>
</div>
</div></div></span>
</div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br>-Debo~<br><br>
</font></span></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br></div></div><div class="im">~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>Dan Wendlandt <div>Nicira, Inc: <a href="http://www.nicira.com" target="_blank">www.nicira.com</a><br>
<div>twitter: danwendlandt<br>
~~~~~~~~~~~~~~~~~~~~~~~~~~~<br></div></div><br>
</div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>Dan Wendlandt <div>Nicira, Inc: <a href="http://www.nicira.com" target="_blank">www.nicira.com</a><br><div>twitter: danwendlandt<br>
~~~~~~~~~~~~~~~~~~~~~~~~~~~<br></div></div><br>
</div>