[openstack-dev] Quantum Performance
Dan Wendlandt
dan at nicira.com
Wed Nov 7 18:53:00 UTC 2012
On Wed, Nov 7, 2012 at 10:28 AM, Juergen Brendel (jbrendel) <
jbrendel at cisco.com> wrote:
> ** **
>
> Dan,****
>
> ** **
>
> Yes, that’s exactly what it’s going to be like. Sorry, I’m new top
> OpenStack development, so I did the fix already, but still need to figure
> out the gerrit side of things.****
>
> ** **
>
> Yes, the code checks for the presence of a specialized count function and
> uses it if the plugin provides it.****
>
> ** **
>
> I hope to have this available sometime today for review.Juergen****
>
No worries, take your time. Below is a proof-of-concept diff that I put
together, in case there are useful things to incorporate into your patch
(e.g., its easy to overlook extended resources that we also do quotas for,
like routers, floatingips, and security_groups).
Dan
nicira at com-dev:/opt/stack/quantum$ git diff
diff --git a/quantum/db/db_base_plugin_v2.py
b/quantum/db/db_base_plugin_v2.py
index a37516e..993bbad 100644
--- a/quantum/db/db_base_plugin_v2.py
+++ b/quantum/db/db_base_plugin_v2.py
@@ -212,6 +212,11 @@ class
QuantumDbPluginV2(quantum_plugin_base_v2.QuantumPluginBaseV
collection = self._apply_filters_to_query(collection, model,
filters)
return [dict_func(c, fields) for c in collection.all()]
+ def _count_collection(self, context, model, filters=None):
+ collection = self._model_query(context, model)
+ collection = self._apply_filters_to_query(collection, model,
filters)
+ return collection.count()
+
@staticmethod
def _generate_mac(context, network_id):
base_mac = cfg.CONF.base_mac.split(':')
@@ -928,6 +933,10 @@ class
QuantumDbPluginV2(quantum_plugin_base_v2.QuantumPluginBaseV
self._make_network_dict,
filters=filters, fields=fields)
+ def count_networks(self, context, filters=None, fields=None):
+ return self._count_collection(context, models_v2.Network,
+ filters=filters)
+
def create_subnet_bulk(self, context, subnets):
return self._create_bulk('subnet', context, subnets)
@@ -1119,6 +1128,10 @@ class
QuantumDbPluginV2(quantum_plugin_base_v2.QuantumPluginBas
self._make_subnet_dict,
filters=filters, fields=fields)
+ def count_subnets(self, context, filters=None, fields=None):
+ return self._count_collection(context, models_v2.Subnet,
+ filters=filters)
+
def create_port_bulk(self, context, ports):
return self._create_bulk('port', context, ports)
@@ -1267,3 +1280,7 @@ class
QuantumDbPluginV2(quantum_plugin_base_v2.QuantumPluginBase
query = self._apply_filters_to_query(query, Port, filters)
return [self._make_port_dict(c, fields) for c in query.all()]
+
+ def count_ports(self, context, filters=None, fields=None):
+ return self._count_collection(context, models_v2.Port,
+ filters=filters)
diff --git a/quantum/db/l3_db.py b/quantum/db/l3_db.py
index f448a68..5318c21 100644
--- a/quantum/db/l3_db.py
+++ b/quantum/db/l3_db.py
@@ -252,6 +252,10 @@ class L3_NAT_db_mixin(l3.RouterPluginBase):
self._make_router_dict,
filters=filters, fields=fields)
+ def count_routers(self, context, filters=None, fields=None):
+ return self._count_collection(context, Router,
+ filters=filters)
+
def _check_for_dup_router_subnet(self, context, router_id,
network_id, subnet_id):
try:
@@ -616,6 +620,10 @@ class L3_NAT_db_mixin(l3.RouterPluginBase):
self._make_floatingip_dict,
filters=filters, fields=fields)
+ def count_floatingips(self, context, filters=None, fields=None):
+ return self._count_collection(context, FloatingIP,
+ filters=filters)
+
def prevent_l3_port_deletion(self, context, port_id):
""" Checks to make sure a port is allowed to be deleted, raising
an exception if this is not the case. This should be called by
diff --git a/quantum/db/securitygroups_db.py
b/quantum/db/securitygroups_db.py
index 2093b33..984d6c3 100644
--- a/quantum/db/securitygroups_db.py
+++ b/quantum/db/securitygroups_db.py
@@ -150,6 +150,9 @@ class
SecurityGroupDbMixin(ext_sg.SecurityGroupPluginBase):
self._make_security_group_dict,
filters=filters, fields=fields)
+ def count_security_groups(self, context, filters=None):
+ return self._count_collection(context, SecurityGroup,
filters=filters)
+
def get_security_group(self, context, id, fields=None, tenant_id=None):
"""Tenant id is given to handle the case when we
are creating a security group or security group rule on behalf of
diff --git a/quantum/quota.py b/quantum/quota.py
index bcb498f..2ebefab 100644
--- a/quantum/quota.py
+++ b/quantum/quota.py
@@ -266,8 +266,13 @@ QUOTAS = QuotaEngine()
def _count_resource(context, plugin, resources, tenant_id):
+ filters = {'tenant_id': [tenant_id]}
+ if hasattr(plugin, "count_%s" % resources):
+ obj_counter = getattr(plugin, "count_%s" % resources)
+ return obj_counter(context, filters=filters)
+
obj_getter = getattr(plugin, "get_%s" % resources)
- obj_list = obj_getter(context, filters={'tenant_id': [tenant_id]})
+ obj_list = obj_getter(context, filters=filters)
return len(obj_list) if obj_list else 0
> ** **
>
> ** **
>
> *From:* Dan Wendlandt [mailto:dan at nicira.com]
> *Sent:* Thursday, November 08, 2012 7:21 AM
>
> *To:* OpenStack Development Mailing List
> *Subject:* Re: [openstack-dev] Quantum Performance****
>
> ** **
>
> ** **
>
> ** **
>
> On Wed, Nov 7, 2012 at 10:12 AM, Dan Wendlandt <dan at nicira.com> wrote:****
>
> I'd like to see something like this automated as part of the smoke tests,
> otherwise its easy for performance regressions to sneak in. I am not aware
> of this existing for a Folsom-based setup yet. ****
>
> ** **
>
> This seems to be the bug we're using to track:
> https://bugs.launchpad.net/quantum/+bug/1075369. I don't see a linked
> review on gerrit though. Have you not yet posted it? ****
>
> ** **
>
> Actually, looking at the code, the change is pretty simple. The only real
> trick is to handle the fact that a true "count_x" resource is not part of
> the plugin API, but would make sense given quotas. Since we can't change
> the plugin API after the fact for Folsom, it probably makes sense to have
> the quota code check if a count_x method exists, and if not, fall back to
> using the existing method. For any plugin that inherits from
> db_base_plugin_v2, we can provide default count_x methods that use
> sqlalchemy.****
>
> ** **
>
> Dan****
>
> ** **
>
> ** **
>
> ****
>
> ** **
>
> Dan****
>
> ** **
>
> ** **
>
> On Wed, Nov 7, 2012 at 7:56 AM, Debojyoti Dutta <ddutta at gmail.com> wrote:*
> ***
>
> ** **
>
> In fact this points to the fact that we need a Geekbench equivalent for
> openstack so that folks can submit performance numbers for their scaled
> setups. ****
>
> ** **
>
> For starters, we could have simple metrics like the test here (for
> quantum) and repeat this for nova too (for starting a bunch of VMs). ****
>
> ** **
>
> The submitted numbers will also be a metric for new fixes/features which
> might have performance sideeffects****
>
> ** **
>
> debo****
>
> ** **
>
> ** **
>
> On Wed, Nov 7, 2012 at 7:20 AM, Edgar Magana (eperdomo) <
> eperdomo at cisco.com> wrote:****
>
> Team,****
>
> ** **
>
> We ran a comparative tests between current Quantum code and our proposed
> fix for this bug and the results are incredible, in current quantum code it
> takes around 4-5 seconds to create a network when already have over 600
> networks created and almost 7 second when we have over 1K networks. See
> this log:****
>
> ** **
>
> Tue Nov 6 23:39:52 PST 2012****
>
> Created a new network:****
>
> +---------------------------+--------------------------------------+****
>
> | Field | Value |****
>
> +---------------------------+--------------------------------------+****
>
> | admin_state_up | True |****
>
> | id | e87b2f27-9308-4001-b180-228ff5ab6479 |****
>
> | name | net621 |****
>
> | provider:network_type | local |****
>
> | provider:physical_network | |****
>
> | provider:segmentation_id | |****
>
> | router:external | False |****
>
> | shared | False |****
>
> | status | ACTIVE |****
>
> | subnets | |****
>
> | tenant_id | b248e18b173b4ea99873e102a0715dcd |****
>
> +---------------------------+--------------------------------------+****
>
> Tue Nov 6 23:39:57 PST 2012****
>
> ++++****
>
> Wed Nov 7 00:17:46 PST 2012****
>
> Created a new network:****
>
> +---------------------------+--------------------------------------+****
>
> | Field | Value |****
>
> +---------------------------+--------------------------------------+****
>
> | admin_state_up | True |****
>
> | id | 2138b910-8996-40b7-a9b4-ac090e126a2b |****
>
> | name | net1032 |****
>
> | provider:network_type | local |****
>
> | provider:physical_network | |****
>
> | provider:segmentation_id | |****
>
> | router:external | False |****
>
> | shared | False |****
>
> | status | ACTIVE |****
>
> | subnets | |****
>
> | tenant_id | b248e18b173b4ea99873e102a0715dcd |****
>
> +---------------------------+--------------------------------------+****
>
> Wed Nov 7 00:17:53 PST 2012****
>
> Wed Nov 7 00:17:53 PST 2012****
>
> Created a new network:****
>
> +---------------------------+--------------------------------------+****
>
> | Field | Value |****
>
> +---------------------------+--------------------------------------+****
>
> | admin_state_up | True |****
>
> | id | 29530fb3-1a1a-4c88-b9e4-954988bb8ccb |****
>
> | name | net1033 |****
>
> | provider:network_type | local |****
>
> | provider:physical_network | |****
>
> | provider:segmentation_id | |****
>
> | router:external | False |****
>
> | shared | False |****
>
> | status | ACTIVE |****
>
> | subnets | |****
>
> | tenant_id | b248e18b173b4ea99873e102a0715dcd |****
>
> +---------------------------+--------------------------------------+****
>
> Wed Nov 7 00:18:00 PST 2012****
>
> ** **
>
> With the changes on the count call we have a constant performance
> regardless the number of networks created:****
>
> ** **
>
> Wed Nov 7 02:48:59 PST 2012****
>
> Created a new network:****
>
> +---------------------------+--------------------------------------+****
>
> | Field | Value |****
>
> +---------------------------+--------------------------------------+****
>
> | admin_state_up | True |****
>
> | id | 024aa164-cb7b-4683-a5ae-41f11a582529 |****
>
> | name | net3847 |****
>
> | provider:network_type | local |****
>
> | provider:physical_network | |****
>
> | provider:segmentation_id | |****
>
> | router:external | False |****
>
> | shared | False |****
>
> | status | ACTIVE |****
>
> | subnets | |****
>
> | tenant_id | 5baba10045db44cdad235fff1d5e59b1 |****
>
> +---------------------------+--------------------------------------+****
>
> Wed Nov 7 02:49:00 PST 2012****
>
> Wed Nov 7 02:49:00 PST 2012****
>
> Created a new network:****
>
> +---------------------------+--------------------------------------+****
>
> | Field | Value |****
>
> +---------------------------+--------------------------------------+****
>
> | admin_state_up | True |****
>
> | id | 91e7eaa3-bdc4-4705-a8b5-2b63500fd066 |****
>
> | name | net3848 |****
>
> | provider:network_type | local |****
>
> | provider:physical_network | |****
>
> | provider:segmentation_id | |****
>
> | router:external | False |****
>
> | shared | False |****
>
> | status | ACTIVE |****
>
> | subnets | |****
>
> | tenant_id | 5baba10045db44cdad235fff1d5e59b1 |****
>
> +---------------------------+--------------------------------------+****
>
> Wed Nov 7 02:49:01 PST 2012****
>
> Wed Nov 7 02:49:01 PST 2012****
>
> Created a new network:****
>
> +---------------------------+--------------------------------------+****
>
> | Field | Value |****
>
> +---------------------------+--------------------------------------+****
>
> | admin_state_up | True |****
>
> | id | 19fd1785-77c2-4526-9fa6-5a2c8c01b3a4 |****
>
> | name | net3849 |****
>
> | provider:network_type | local |****
>
> | provider:physical_network | |****
>
> | provider:segmentation_id | |****
>
> | router:external | False |****
>
> | shared | False |****
>
> | status | ACTIVE |****
>
> | subnets | |****
>
> | tenant_id | 5baba10045db44cdad235fff1d5e59b1 |****
>
> +---------------------------+--------------------------------------+****
>
> Wed Nov 7 02:49:01 PST 2012****
>
> ** **
>
> We will push this code changes ASAP.****
>
> ** **
>
> Thanks,****
>
> ** **
>
> Edgar****
>
> ** **
>
> *From: *Edgar Magana <eperdomo at cisco.com>
> *Reply-To: *OpenStack List <openstack-dev at lists.openstack.org>
> *Date: *Tuesday, November 6, 2012 9:19 AM
> *To: *OpenStack List <openstack-dev at lists.openstack.org>, "
> gkotton at redhat.com" <gkotton at redhat.com>****
>
>
> *Subject: *Re: [openstack-dev] Quantum Performance****
>
> ** **
>
> Hi Folks,****
>
> ** **
>
> I just want to give you an update on this topic, we opened a bug against
> this behavior:****
>
> https://bugs.launchpad.net/quantum/+bug/1075369****
>
> ** **
>
> Bug Description:****
>
> In quantum when a new network is created and system checks for user's
> quota, instead of getting a count from DB object, it returns all objects
> from db and locally counts them all. It has performance implications when
> the number of objects increases.****
>
> ** **
>
> We will submit the patch ASAP.****
>
> ** **
>
> Thanks,****
>
> ** **
>
> Edgar****
>
> ** **
>
> *From: *Salvatore Orlando <sorlando at nicira.com>
> *Reply-To: *OpenStack List <openstack-dev at lists.openstack.org>
> *Date: *Wednesday, October 31, 2012 12:54 AM
> *To: *"gkotton at redhat.com" <gkotton at redhat.com>, OpenStack List <
> openstack-dev at lists.openstack.org>
> *Subject: *Re: [openstack-dev] Quantum Performance****
>
> ** **
>
> Edgar, ****
>
> ** **
>
> My suspicion here goes to the IP allocation mechanisms.****
>
> Hence, I'd try to run the following tests (they should be easily
> scriptable):****
>
> ** **
>
> 1) Create networks no subnets****
>
> 2) Create networks, with subnet having a small CIDR (say /28)****
>
> 3) Create networks with larger CIDRs (say /24, /20, /16 etc)****
>
> ** **
>
> What behaviour do you observe in the above three cases?****
>
> ** **
>
> Also, It might be worth shutting down agent to pinpoint whether the
> problem is in Quantum service itself on in the interactions with the
> various agents.****
>
> ** **
>
> Salvatore****
>
> ** **
>
> PS: thanks for doing these tests - They've been lying on my TODO list for
> way too much time!****
>
> If you don't hate me too much for the service insertion story, I'll be
> glad to buy you a drink at the next summit :)****
>
> ** **
>
> ** **
>
> On 31 October 2012 08:37, Gary Kotton <gkotton at redhat.com> wrote:****
>
> On 10/31/2012 09:48 AM, Edgar Magana (eperdomo) wrote: ****
>
> Hi Folks,****
>
> ** **
>
> I am running OpenStack Folsom with Quantum/OVS plugin. I am running some
> performance tests for Quantum, basically I am creating 5000 network with
> one subnet each one of them.****
>
> After the first ~50 – 70 networks the system response time is slower and
> slower, to the point that it could take up to ~6 - 7 seconds to create just
> a network and the same for the subnets.****
>
> ** **
>
> Hi,
> Is the problem you are seeing with the Quantum service or the agents? Off
> the bat I would say that it is with the Quantum service. Are the calls done
> in parallel or are they done sequentially?
>
> There are a number of things that we should do to isolate this:
> 1. Check the internal logic of the network creation. Basically there are 3
> stages:
> i. provider network treatment
> ii network create
> iii. l3 notification
> It would be interesting to see who eats the most time.
> 2. Need to profile database access
>
> Thanks
> Gary
>
>
> ****
>
> I would like to know if somebody else has experimented this kind of
> behavior in Quantum or even in nova-network.****
>
> ** **
>
> Thanks,****
>
> ** **
>
> Edgar****
>
> ****
>
> ** **
>
> _______________________________________________****
>
> OpenStack-dev mailing list****
>
> OpenStack-dev at lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev****
>
> ** **
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev****
>
> ** **
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev****
>
>
>
> ****
>
> ** **
>
> --
> -Debo~****
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev****
>
>
>
> ****
>
> ** **
>
> -- ****
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Dan Wendlandt ****
>
> Nicira, Inc: www.nicira.com****
>
> twitter: danwendlandt
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~****
>
> ** **
>
>
>
> ****
>
> ** **
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Dan Wendlandt ****
>
> Nicira, Inc: www.nicira.com****
>
> twitter: danwendlandt
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~****
>
> ** **
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira, Inc: www.nicira.com
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121107/6706d324/attachment-0001.html>
More information about the OpenStack-dev
mailing list