[openstack-dev] [cinder] [manila] Performance concern on new quota system

Gorka Eguileor geguileo at redhat.com
Fri Mar 9 12:11:02 UTC 2018


On 09/03, TommyLike Hu wrote:
> Hey all,
>     During the Cinder and Manila's cross project discussion on quota system
> last week, I raise our concern of performance impact on the count resource
> feature, and Gorka pointed out we might miss some of the indexes when
> testing. So I reshaped the environment and share some test results here as
> below:
>
> *Test patch:*
>      Basically this [1] is borrowed from Nova for Cinder, and only the
> create volume process has been updated to use the new system.
>
> *Test Environment on AWS:*
>     1. 3 EC2 t2.large(2 vCPU, 8GiB,80G SSD) each one deployed cinder API
> service with 10 api workers (coordinated by haproxy)
>     2. 1 RDS db.m4.xlarge(4 vCPU, 16GiB,200G SSD) MySQL 5.7.19
>
> *Database upgrade:*
> Composite index has been added to volume table (volumes__composite_index):
> +---------+------------+--------------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
> | Table   | Non_unique | Key_name                      | Seq_in_index |
> Column_name         | Collation | Cardinality | Sub_part | Packed | Null |
> Index_type | Comment | Index_comment |
> +---------+------------+--------------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
> |   .....other index.......     |
> | volumes |                 1 | volumes__composite_index |             1 |
> project_id          | A         |           2 |     NULL | NULL   | YES  |
> BTREE      |         |               |
> | volumes |                 1 | volumes__compoite_index |               2 |
> deleted             | A         |           2 |     NULL | NULL   | YES  |
> BTREE      |         |               |
> | volumes |                 1 | volumes__composite_index |             3 |
> volume_type_id      | A         |           2 |     NULL | NULL   | YES  |
> BTREE      |         |               |
> +---------+------------+--------------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
>  Explain result for one of the sql statements for new quota system (*explain
> select count(*), sum(size) from volumes where project_id={project_id} and
> volume_type_id={type_id} and deleted=false*):
>
>  +----+-------------+---------+------------+------+--------------------------+--------------------------+---------+-------------------+------+----------+-------+
> | id | select_type | table   | partitions | type | possible_keys
>  | key                      | key_len | ref               | rows | filtered
> | Extra |
>
>  +----+-------------+---------+------------+------+--------------------------+--------------------------+---------+-------------------+------+----------+-------+
> |  1 | SIMPLE      | volumes | NULL       | ref  | volumes__composite_index
> | volumes__composite_index | 881     | const,const,const |    1 |   100.00
> | NULL  |
>
>  +----+-------------+---------+------------+------+--------------------------+--------------------------+---------+-------------------+------+----------+-------+
>
> *Time comparsion between two system (in Seconds, * *Conc = Concurrency**):*
>
> *NOTE**:*
>
>       1. *QUOTAS.check_deltas*  stands for the total time consumed
> including two sql statements as below when creating single volume:
>
>           1. SELECT count(id), sum(size) from volumes where
> project_id=$(project_id) and deleted=False
>
>           2. SELECT count(id), sum(size) from volumes where
> project_id=$(project_id) and deleted=False and volume_type=$(volume_type)
>

Hi,

As I see it there are 3 questions here:

1. Do we need to change the quota system?

   I believe we all agree that the Quota system in Cinder is bad, so bad
   that we can divide Cinder deployments in 2 clear categories, those
   that don't use quotas and those that have out of sync quotas.

2. Will the DB implementation used by Nova solve our current problems?

   As the Nova team kindly explained us, the new quotas may not be
   perfect for limiting (we may allow going slightly above allowed quota
   or just get short), but it will at least solve our problems of having
   out of sync quotas that require manually changing the DB to fix it.

3. Will the new solution introduce new problems?

   To me introducing a small performance impact on resource creation is
   an acceptable trade-off compared with the alternative, moreover
   considering that most resource creation procedures are somewhat slow
   operations.

   And let's not forget that we can always look for more efficient ways
   of doing the counting:

   - Using a single query to retrieve both counts and sums instead of 2
     queries.

   - DB triggers to do the actual counting.

To me comparing the performance of something that doesn't work with
something that does doesn't seem fair.

Cheers,
Gorka.



>       2. *Quota Reserve/Quota Commit* stands for total time consumed when
> executing QUOTA.reserve and QUOTA.commit.
>
> 1. Create 1000 volumes in tenant which has *10000* records in database
> and *180000
> *undeleted records in total:
> [image: image.png]
> 2.  Create 1000 volumes in tenant which has *20000* records in
> database and *180000
> *undeleted records in total:
>
> [image: image.png]
> 3.  Create 1000 volumes in tenant which has *30000* records in
> database and *180000
> *undeleted records in total:
> [image: image.png]
> 4.  Create 1000 volumes in tenant which has* 40000* records in
> database and *180000
> *undeleted records in total:
>
> [image: image.png]
> 5.  Create 1000 volumes in tenant which has* 60000* records in
> database and *180000
> *undeleted records in total:
> [image: image.png]
> I only posted some of the test results here, but in general, the new system
> will become slower when the amount of concurrency or existing volumes in
> tenant keeps raising.  Also it seems our current quota system will always
> beat the new one in performance when there are about *30000 *volumes in
> tenant.
>
> I am a little worried about the performance impact if we replace our
> current design with count resource feature, and I could be wrong, maybe I
> missed something important during testing, please let me know if you have
> any idea or suggestion.
>
> Thanks
> TommyLike
>
> [1]: https://review.openstack.org/#/c/536341/







> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




More information about the OpenStack-dev mailing list