[openstack-dev] Quantum Performance
Dan Wendlandt
dan at nicira.com
Wed Nov 7 18:12:11 UTC 2012
I'd like to see something like this automated as part of the smoke tests,
otherwise its easy for performance regressions to sneak in. I am not aware
of this existing for a Folsom-based setup yet.
This seems to be the bug we're using to track:
https://bugs.launchpad.net/quantum/+bug/1075369. I don't see a linked
review on gerrit though. Have you not yet posted it?
Dan
On Wed, Nov 7, 2012 at 7:56 AM, Debojyoti Dutta <ddutta at gmail.com> wrote:
>
> In fact this points to the fact that we need a Geekbench equivalent for
> openstack so that folks can submit performance numbers for their scaled
> setups.
>
> For starters, we could have simple metrics like the test here (for
> quantum) and repeat this for nova too (for starting a bunch of VMs).
>
> The submitted numbers will also be a metric for new fixes/features which
> might have performance sideeffects
>
> debo
>
>
>
> On Wed, Nov 7, 2012 at 7:20 AM, Edgar Magana (eperdomo) <
> eperdomo at cisco.com> wrote:
>
>> Team,
>>
>> We ran a comparative tests between current Quantum code and our
>> proposed fix for this bug and the results are incredible, in current
>> quantum code it takes around 4-5 seconds to create a network when already
>> have over 600 networks created and almost 7 second when we have over 1K
>> networks. See this log:
>>
>> Tue Nov 6 23:39:52 PST 2012
>> Created a new network:
>> +---------------------------+--------------------------------------+
>> | Field | Value |
>> +---------------------------+--------------------------------------+
>> | admin_state_up | True |
>> | id | e87b2f27-9308-4001-b180-228ff5ab6479 |
>> | name | net621 |
>> | provider:network_type | local |
>> | provider:physical_network | |
>> | provider:segmentation_id | |
>> | router:external | False |
>> | shared | False |
>> | status | ACTIVE |
>> | subnets | |
>> | tenant_id | b248e18b173b4ea99873e102a0715dcd |
>> +---------------------------+--------------------------------------+
>> Tue Nov 6 23:39:57 PST 2012
>> ++++
>> Wed Nov 7 00:17:46 PST 2012
>> Created a new network:
>> +---------------------------+--------------------------------------+
>> | Field | Value |
>> +---------------------------+--------------------------------------+
>> | admin_state_up | True |
>> | id | 2138b910-8996-40b7-a9b4-ac090e126a2b |
>> | name | net1032 |
>> | provider:network_type | local |
>> | provider:physical_network | |
>> | provider:segmentation_id | |
>> | router:external | False |
>> | shared | False |
>> | status | ACTIVE |
>> | subnets | |
>> | tenant_id | b248e18b173b4ea99873e102a0715dcd |
>> +---------------------------+--------------------------------------+
>> Wed Nov 7 00:17:53 PST 2012
>> Wed Nov 7 00:17:53 PST 2012
>> Created a new network:
>> +---------------------------+--------------------------------------+
>> | Field | Value |
>> +---------------------------+--------------------------------------+
>> | admin_state_up | True |
>> | id | 29530fb3-1a1a-4c88-b9e4-954988bb8ccb |
>> | name | net1033 |
>> | provider:network_type | local |
>> | provider:physical_network | |
>> | provider:segmentation_id | |
>> | router:external | False |
>> | shared | False |
>> | status | ACTIVE |
>> | subnets | |
>> | tenant_id | b248e18b173b4ea99873e102a0715dcd |
>> +---------------------------+--------------------------------------+
>> Wed Nov 7 00:18:00 PST 2012
>>
>> With the changes on the count call we have a constant performance
>> regardless the number of networks created:
>>
>> Wed Nov 7 02:48:59 PST 2012
>> Created a new network:
>> +---------------------------+--------------------------------------+
>> | Field | Value |
>> +---------------------------+--------------------------------------+
>> | admin_state_up | True |
>> | id | 024aa164-cb7b-4683-a5ae-41f11a582529 |
>> | name | net3847 |
>> | provider:network_type | local |
>> | provider:physical_network | |
>> | provider:segmentation_id | |
>> | router:external | False |
>> | shared | False |
>> | status | ACTIVE |
>> | subnets | |
>> | tenant_id | 5baba10045db44cdad235fff1d5e59b1 |
>> +---------------------------+--------------------------------------+
>> Wed Nov 7 02:49:00 PST 2012
>> Wed Nov 7 02:49:00 PST 2012
>> Created a new network:
>> +---------------------------+--------------------------------------+
>> | Field | Value |
>> +---------------------------+--------------------------------------+
>> | admin_state_up | True |
>> | id | 91e7eaa3-bdc4-4705-a8b5-2b63500fd066 |
>> | name | net3848 |
>> | provider:network_type | local |
>> | provider:physical_network | |
>> | provider:segmentation_id | |
>> | router:external | False |
>> | shared | False |
>> | status | ACTIVE |
>> | subnets | |
>> | tenant_id | 5baba10045db44cdad235fff1d5e59b1 |
>> +---------------------------+--------------------------------------+
>> Wed Nov 7 02:49:01 PST 2012
>> Wed Nov 7 02:49:01 PST 2012
>> Created a new network:
>> +---------------------------+--------------------------------------+
>> | Field | Value |
>> +---------------------------+--------------------------------------+
>> | admin_state_up | True |
>> | id | 19fd1785-77c2-4526-9fa6-5a2c8c01b3a4 |
>> | name | net3849 |
>> | provider:network_type | local |
>> | provider:physical_network | |
>> | provider:segmentation_id | |
>> | router:external | False |
>> | shared | False |
>> | status | ACTIVE |
>> | subnets | |
>> | tenant_id | 5baba10045db44cdad235fff1d5e59b1 |
>> +---------------------------+--------------------------------------+
>> Wed Nov 7 02:49:01 PST 2012
>>
>> We will push this code changes ASAP.
>>
>> Thanks,
>>
>> Edgar
>>
>> From: Edgar Magana <eperdomo at cisco.com>
>> Reply-To: OpenStack List <openstack-dev at lists.openstack.org>
>> Date: Tuesday, November 6, 2012 9:19 AM
>> To: OpenStack List <openstack-dev at lists.openstack.org>, "
>> gkotton at redhat.com" <gkotton at redhat.com>
>>
>> Subject: Re: [openstack-dev] Quantum Performance
>>
>> Hi Folks,
>>
>> I just want to give you an update on this topic, we opened a bug
>> against this behavior:
>> https://bugs.launchpad.net/quantum/+bug/1075369
>>
>> Bug Description:
>> In quantum when a new network is created and system checks for user's
>> quota, instead of getting a count from DB object, it returns all objects
>> from db and locally counts them all. It has performance implications when
>> the number of objects increases.
>>
>> We will submit the patch ASAP.
>>
>> Thanks,
>>
>> Edgar
>>
>> From: Salvatore Orlando <sorlando at nicira.com>
>> Reply-To: OpenStack List <openstack-dev at lists.openstack.org>
>> Date: Wednesday, October 31, 2012 12:54 AM
>> To: "gkotton at redhat.com" <gkotton at redhat.com>, OpenStack List <
>> openstack-dev at lists.openstack.org>
>> Subject: Re: [openstack-dev] Quantum Performance
>>
>> Edgar,
>>
>> My suspicion here goes to the IP allocation mechanisms.
>> Hence, I'd try to run the following tests (they should be easily
>> scriptable):
>>
>> 1) Create networks no subnets
>> 2) Create networks, with subnet having a small CIDR (say /28)
>> 3) Create networks with larger CIDRs (say /24, /20, /16 etc)
>>
>> What behaviour do you observe in the above three cases?
>>
>> Also, It might be worth shutting down agent to pinpoint whether the
>> problem is in Quantum service itself on in the interactions with the
>> various agents.
>>
>> Salvatore
>>
>> PS: thanks for doing these tests - They've been lying on my TODO list
>> for way too much time!
>> If you don't hate me too much for the service insertion story, I'll be
>> glad to buy you a drink at the next summit :)
>>
>>
>> On 31 October 2012 08:37, Gary Kotton <gkotton at redhat.com> wrote:
>>
>>> On 10/31/2012 09:48 AM, Edgar Magana (eperdomo) wrote:
>>>
>>> Hi Folks,
>>>
>>> I am running OpenStack Folsom with Quantum/OVS plugin. I am running
>>> some performance tests for Quantum, basically I am creating 5000 network
>>> with one subnet each one of them.
>>> After the first ~50 – 70 networks the system response time is slower and
>>> slower, to the point that it could take up to ~6 - 7 seconds to create just
>>> a network and the same for the subnets.
>>>
>>>
>>> Hi,
>>> Is the problem you are seeing with the Quantum service or the agents?
>>> Off the bat I would say that it is with the Quantum service. Are the calls
>>> done in parallel or are they done sequentially?
>>>
>>> There are a number of things that we should do to isolate this:
>>> 1. Check the internal logic of the network creation. Basically there are
>>> 3 stages:
>>> i. provider network treatment
>>> ii network create
>>> iii. l3 notification
>>> It would be interesting to see who eats the most time.
>>> 2. Need to profile database access
>>>
>>> Thanks
>>> Gary
>>>
>>> I would like to know if somebody else has experimented this kind of
>>> behavior in Quantum or even in nova-network.
>>>
>>> Thanks,
>>>
>>> Edgar
>>>
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing listOpenStack-dev at lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> -Debo~
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira, Inc: www.nicira.com
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121107/a299ab6f/attachment.html>
More information about the OpenStack-dev
mailing list