[openstack-dev] Quantum Performance
Edgar Magana (eperdomo)
eperdomo at cisco.com
Wed Nov 7 15:20:17 UTC 2012
Team,
We ran a comparative tests between current Quantum code and our proposed fix for this bug and the results are incredible, in current quantum code it takes around 4-5 seconds to create a network when already have over 600 networks created and almost 7 second when we have over 1K networks. See this log:
Tue Nov 6 23:39:52 PST 2012
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | e87b2f27-9308-4001-b180-228ff5ab6479 |
| name | net621 |
| provider:network_type | local |
| provider:physical_network | |
| provider:segmentation_id | |
| router:external | False |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | b248e18b173b4ea99873e102a0715dcd |
+---------------------------+--------------------------------------+
Tue Nov 6 23:39:57 PST 2012
++++
Wed Nov 7 00:17:46 PST 2012
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 2138b910-8996-40b7-a9b4-ac090e126a2b |
| name | net1032 |
| provider:network_type | local |
| provider:physical_network | |
| provider:segmentation_id | |
| router:external | False |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | b248e18b173b4ea99873e102a0715dcd |
+---------------------------+--------------------------------------+
Wed Nov 7 00:17:53 PST 2012
Wed Nov 7 00:17:53 PST 2012
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 29530fb3-1a1a-4c88-b9e4-954988bb8ccb |
| name | net1033 |
| provider:network_type | local |
| provider:physical_network | |
| provider:segmentation_id | |
| router:external | False |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | b248e18b173b4ea99873e102a0715dcd |
+---------------------------+--------------------------------------+
Wed Nov 7 00:18:00 PST 2012
With the changes on the count call we have a constant performance regardless the number of networks created:
Wed Nov 7 02:48:59 PST 2012
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 024aa164-cb7b-4683-a5ae-41f11a582529 |
| name | net3847 |
| provider:network_type | local |
| provider:physical_network | |
| provider:segmentation_id | |
| router:external | False |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | 5baba10045db44cdad235fff1d5e59b1 |
+---------------------------+--------------------------------------+
Wed Nov 7 02:49:00 PST 2012
Wed Nov 7 02:49:00 PST 2012
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 91e7eaa3-bdc4-4705-a8b5-2b63500fd066 |
| name | net3848 |
| provider:network_type | local |
| provider:physical_network | |
| provider:segmentation_id | |
| router:external | False |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | 5baba10045db44cdad235fff1d5e59b1 |
+---------------------------+--------------------------------------+
Wed Nov 7 02:49:01 PST 2012
Wed Nov 7 02:49:01 PST 2012
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 19fd1785-77c2-4526-9fa6-5a2c8c01b3a4 |
| name | net3849 |
| provider:network_type | local |
| provider:physical_network | |
| provider:segmentation_id | |
| router:external | False |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | 5baba10045db44cdad235fff1d5e59b1 |
+---------------------------+--------------------------------------+
Wed Nov 7 02:49:01 PST 2012
We will push this code changes ASAP.
Thanks,
Edgar
From: Edgar Magana <eperdomo at cisco.com<mailto:eperdomo at cisco.com>>
Reply-To: OpenStack List <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Tuesday, November 6, 2012 9:19 AM
To: OpenStack List <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>, "gkotton at redhat.com<mailto:gkotton at redhat.com>" <gkotton at redhat.com<mailto:gkotton at redhat.com>>
Subject: Re: [openstack-dev] Quantum Performance
Hi Folks,
I just want to give you an update on this topic, we opened a bug against this behavior:
https://bugs.launchpad.net/quantum/+bug/1075369
Bug Description:
In quantum when a new network is created and system checks for user's quota, instead of getting a count from DB object, it returns all objects from db and locally counts them all. It has performance implications when the number of objects increases.
We will submit the patch ASAP.
Thanks,
Edgar
From: Salvatore Orlando <sorlando at nicira.com<mailto:sorlando at nicira.com>>
Reply-To: OpenStack List <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Wednesday, October 31, 2012 12:54 AM
To: "gkotton at redhat.com<mailto:gkotton at redhat.com>" <gkotton at redhat.com<mailto:gkotton at redhat.com>>, OpenStack List <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Subject: Re: [openstack-dev] Quantum Performance
Edgar,
My suspicion here goes to the IP allocation mechanisms.
Hence, I'd try to run the following tests (they should be easily scriptable):
1) Create networks no subnets
2) Create networks, with subnet having a small CIDR (say /28)
3) Create networks with larger CIDRs (say /24, /20, /16 etc)
What behaviour do you observe in the above three cases?
Also, It might be worth shutting down agent to pinpoint whether the problem is in Quantum service itself on in the interactions with the various agents.
Salvatore
PS: thanks for doing these tests - They've been lying on my TODO list for way too much time!
If you don't hate me too much for the service insertion story, I'll be glad to buy you a drink at the next summit :)
On 31 October 2012 08:37, Gary Kotton <gkotton at redhat.com<mailto:gkotton at redhat.com>> wrote:
On 10/31/2012 09:48 AM, Edgar Magana (eperdomo) wrote:
Hi Folks,
I am running OpenStack Folsom with Quantum/OVS plugin. I am running some performance tests for Quantum, basically I am creating 5000 network with one subnet each one of them.
After the first ~50 – 70 networks the system response time is slower and slower, to the point that it could take up to ~6 - 7 seconds to create just a network and the same for the subnets.
Hi,
Is the problem you are seeing with the Quantum service or the agents? Off the bat I would say that it is with the Quantum service. Are the calls done in parallel or are they done sequentially?
There are a number of things that we should do to isolate this:
1. Check the internal logic of the network creation. Basically there are 3 stages:
i. provider network treatment
ii network create
iii. l3 notification
It would be interesting to see who eats the most time.
2. Need to profile database access
Thanks
Gary
I would like to know if somebody else has experimented this kind of behavior in Quantum or even in nova-network.
Thanks,
Edgar
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121107/a9143a54/attachment.html>
More information about the OpenStack-dev
mailing list