<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>Team,</div>
<div><br>
</div>
<div>We ran a comparative tests between current Quantum code and our proposed fix for this bug and the results are incredible, in current quantum code it takes around 4-5 seconds to create a network when already have over 600 networks created and almost 7 second
when we have over 1K networks. See this log:</div>
<div><br>
</div>
<div>
<div>Tue Nov 6 23:39:52 PST 2012</div>
<div>Created a new network:</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| admin_state_up | True |</div>
<div>| id | e87b2f27-9308-4001-b180-228ff5ab6479 |</div>
<div>| name | net621 |</div>
<div>| provider:network_type | local |</div>
<div>| provider:physical_network | |</div>
<div>| provider:segmentation_id | |</div>
<div>| router:external | False |</div>
<div>| shared | False |</div>
<div>| status | ACTIVE |</div>
<div>| subnets | |</div>
<div>| tenant_id | b248e18b173b4ea99873e102a0715dcd |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>Tue Nov 6 23:39:57 PST 2012</div>
</div>
<div>++++</div>
<div>
<div>Wed Nov 7 00:17:46 PST 2012</div>
<div>Created a new network:</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| admin_state_up | True |</div>
<div>| id | 2138b910-8996-40b7-a9b4-ac090e126a2b |</div>
<div>| name | net1032 |</div>
<div>| provider:network_type | local |</div>
<div>| provider:physical_network | |</div>
<div>| provider:segmentation_id | |</div>
<div>| router:external | False |</div>
<div>| shared | False |</div>
<div>| status | ACTIVE |</div>
<div>| subnets | |</div>
<div>| tenant_id | b248e18b173b4ea99873e102a0715dcd |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>Wed Nov 7 00:17:53 PST 2012</div>
<div>Wed Nov 7 00:17:53 PST 2012</div>
<div>Created a new network:</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| admin_state_up | True |</div>
<div>| id | 29530fb3-1a1a-4c88-b9e4-954988bb8ccb |</div>
<div>| name | net1033 |</div>
<div>| provider:network_type | local |</div>
<div>| provider:physical_network | |</div>
<div>| provider:segmentation_id | |</div>
<div>| router:external | False |</div>
<div>| shared | False |</div>
<div>| status | ACTIVE |</div>
<div>| subnets | |</div>
<div>| tenant_id | b248e18b173b4ea99873e102a0715dcd |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>Wed Nov 7 00:18:00 PST 2012</div>
</div>
<div><br>
</div>
<div>With the changes on the count call we have a constant performance regardless the number of networks created:</div>
<div><br>
</div>
<div>
<div>Wed Nov 7 02:48:59 PST 2012</div>
<div>Created a new network:</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| admin_state_up | True |</div>
<div>| id | 024aa164-cb7b-4683-a5ae-41f11a582529 |</div>
<div>| name | net3847 |</div>
<div>| provider:network_type | local |</div>
<div>| provider:physical_network | |</div>
<div>| provider:segmentation_id | |</div>
<div>| router:external | False |</div>
<div>| shared | False |</div>
<div>| status | ACTIVE |</div>
<div>| subnets | |</div>
<div>| tenant_id | 5baba10045db44cdad235fff1d5e59b1 |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>Wed Nov 7 02:49:00 PST 2012</div>
<div>Wed Nov 7 02:49:00 PST 2012</div>
<div>Created a new network:</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| admin_state_up | True |</div>
<div>| id | 91e7eaa3-bdc4-4705-a8b5-2b63500fd066 |</div>
<div>| name | net3848 |</div>
<div>| provider:network_type | local |</div>
<div>| provider:physical_network | |</div>
<div>| provider:segmentation_id | |</div>
<div>| router:external | False |</div>
<div>| shared | False |</div>
<div>| status | ACTIVE |</div>
<div>| subnets | |</div>
<div>| tenant_id | 5baba10045db44cdad235fff1d5e59b1 |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>Wed Nov 7 02:49:01 PST 2012</div>
<div>Wed Nov 7 02:49:01 PST 2012</div>
<div>Created a new network:</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>| admin_state_up | True |</div>
<div>| id | 19fd1785-77c2-4526-9fa6-5a2c8c01b3a4 |</div>
<div>| name | net3849 |</div>
<div>| provider:network_type | local |</div>
<div>| provider:physical_network | |</div>
<div>| provider:segmentation_id | |</div>
<div>| router:external | False |</div>
<div>| shared | False |</div>
<div>| status | ACTIVE |</div>
<div>| subnets | |</div>
<div>| tenant_id | 5baba10045db44cdad235fff1d5e59b1 |</div>
<div>+---------------------------+--------------------------------------+</div>
<div>Wed Nov 7 02:49:01 PST 2012</div>
</div>
<div><br>
</div>
<div>We will push this code changes ASAP.</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Edgar</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Edgar Magana <<a href="mailto:eperdomo@cisco.com">eperdomo@cisco.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>OpenStack List <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Tuesday, November 6, 2012 9:19 AM<br>
<span style="font-weight:bold">To: </span>OpenStack List <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>>, "<a href="mailto:gkotton@redhat.com">gkotton@redhat.com</a>" <<a href="mailto:gkotton@redhat.com">gkotton@redhat.com</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] Quantum Performance<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
Hi Folks,</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
I just want to give you an update on this topic, we opened a bug against this behavior:</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<a href="https://bugs.launchpad.net/quantum/+bug/1075369">https://bugs.launchpad.net/quantum/+bug/1075369</a></div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<br>
</div>
<div><span style="line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">Bug Description:</span></div>
<div><span style="line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">In quantum when a new network is created and system checks for user's quota, instead of getting a count from DB object, it returns all objects from db and locally
counts them all. It has performance implications when the number of objects increases.</span></div>
<div><span style="line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); "><br>
</span></div>
<div><span style="line-height: 18px; text-align: left; background-color: rgb(255, 254, 254);">We will submit the patch ASAP.</span></div>
<div><span style="line-height: 18px; text-align: left; background-color: rgb(255, 254, 254);"><br>
</span></div>
<div><span style="line-height: 18px; text-align: left; background-color: rgb(254, 253, 253);">Thanks,</span></div>
<div><span style="line-height: 18px; text-align: left; background-color: rgb(254, 253, 253);"><br>
</span></div>
<div><span style="line-height: 18px; text-align: left; background-color: rgb(253, 252, 252);">Edgar</span></div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<br>
</div>
<span id="OLK_SRC_BODY_SECTION" style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Salvatore Orlando <<a href="mailto:sorlando@nicira.com">sorlando@nicira.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>OpenStack List <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, October 31, 2012 12:54 AM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:gkotton@redhat.com">gkotton@redhat.com</a>" <<a href="mailto:gkotton@redhat.com">gkotton@redhat.com</a>>, OpenStack List <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] Quantum Performance<br>
</div>
<div><br>
</div>
<div>
<div>Edgar,
<div><br>
</div>
<div>My suspicion here goes to the IP allocation mechanisms.</div>
<div>Hence, I'd try to run the following tests (they should be easily scriptable):</div>
<div><br>
</div>
<div>1) Create networks no subnets</div>
<div>2) Create networks, with subnet having a small CIDR (say /28)</div>
<div>3) Create networks with larger CIDRs (say /24, /20, /16 etc)</div>
<div><br>
</div>
<div>What behaviour do you observe in the above three cases?</div>
<div><br>
</div>
<div>Also, It might be worth shutting down agent to pinpoint whether the problem is in Quantum service itself on in the interactions with the various agents.</div>
<div><br>
</div>
<div>Salvatore</div>
<div>
<div><br>
</div>
<div>PS: thanks for doing these tests - They've been lying on my TODO list for way too much time!</div>
<div>If you don't hate me too much for the service insertion story, I'll be glad to buy you a drink at the next summit :)</div>
<div><br>
</div>
<br>
<div class="gmail_quote">On 31 October 2012 08:37, Gary Kotton <span dir="ltr"><<a href="mailto:gkotton@redhat.com" target="_blank">gkotton@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div class="im">On 10/31/2012 09:48 AM, Edgar Magana (eperdomo) wrote:
<blockquote type="cite">
<div>Hi Folks,</div>
<div><br>
</div>
<div>I am running OpenStack Folsom with Quantum/OVS plugin. I am running some performance tests for Quantum, basically I am creating 5000 network with one subnet each one of them.</div>
<div>After the first ~50 – 70 networks the system response time is slower and slower, to the point that it could take up to ~6 - 7 seconds to create just a network and the same for the subnets.</div>
</blockquote>
<br>
</div>
Hi,<br>
Is the problem you are seeing with the Quantum service or the agents? Off the bat I would say that it is with the Quantum service. Are the calls done in parallel or are they done sequentially?<br>
<br>
There are a number of things that we should do to isolate this:<br>
1. Check the internal logic of the network creation. Basically there are 3 stages:<br>
i. provider network treatment<br>
ii network create<br>
iii. l3 notification<br>
It would be interesting to see who eats the most time.<br>
2. Need to profile database access<br>
<br>
Thanks<br>
Gary<br>
<br>
<blockquote type="cite">
<div class="im">
<div>I would like to know if somebody else has experimented this kind of behavior in Quantum or even in nova-network.</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Edgar</div>
<div> </div>
<br>
<fieldset></fieldset> <br>
</div>
<pre>_______________________________________________
OpenStack-dev mailing list
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></pre>
</blockquote>
<br>
</div>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</span></div>
</div>
</span>
</body>
</html>