Open Stack

Hey all,
    During the Cinder and Manila's cross project discussion on quota system
last week, I raise our concern of performance impact on the count resource
feature, and Gorka pointed out we might miss some of the indexes when
testing. So I reshaped the environment and share some test results here as
below:

*Test patch:*
     Basically this [1] is borrowed from Nova for Cinder, and only the
create volume process has been updated to use the new system.

*Test Environment on AWS:*
    1. 3 EC2 t2.large(2 vCPU, 8GiB，80G SSD) each one deployed cinder API
service with 10 api workers (coordinated by haproxy)
    2. 1 RDS db.m4.xlarge(4 vCPU, 16GiB，200G SSD) MySQL 5.7.19

*Database upgrade:*
Composite index has been added to volume table (volumes__composite_index):
+---------+------------+--------------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table   | Non_unique | Key_name                      | Seq_in_index |
Column_name         | Collation | Cardinality | Sub_part | Packed | Null |
Index_type | Comment | Index_comment |
+---------+------------+--------------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
|   .....other index.......     |
| volumes |                 1 | volumes__composite_index |             1 |
project_id          | A         |           2 |     NULL | NULL   | YES  |
BTREE      |         |               |
| volumes |                 1 | volumes__compoite_index |               2 |
deleted             | A         |           2 |     NULL | NULL   | YES  |
BTREE      |         |               |
| volumes |                 1 | volumes__composite_index |             3 |
volume_type_id      | A         |           2 |     NULL | NULL   | YES  |
BTREE      |         |               |
+---------+------------+--------------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
 Explain result for one of the sql statements for new quota system (*explain
select count(*), sum(size) from volumes where project_id={project_id} and
volume_type_id={type_id} and deleted=false*):

 +----+-------------+---------+------------+------+--------------------------+--------------------------+---------+-------------------+------+----------+-------+
| id | select_type | table   | partitions | type | possible_keys
 | key                      | key_len | ref               | rows | filtered
| Extra |

 +----+-------------+---------+------------+------+--------------------------+--------------------------+---------+-------------------+------+----------+-------+
|  1 | SIMPLE      | volumes | NULL       | ref  | volumes__composite_index
| volumes__composite_index | 881     | const,const,const |    1 |   100.00
| NULL  |

 +----+-------------+---------+------------+------+--------------------------+--------------------------+---------+-------------------+------+----------+-------+

*Time comparsion between two system (in Seconds, * *Conc = Concurrency**):*

*NOTE**:*

      1. *QUOTAS.check_deltas*  stands for the total time consumed
including two sql statements as below when creating single volume:

          1. SELECT count(id), sum(size) from volumes where
project_id=$(project_id) and deleted=False

          2. SELECT count(id), sum(size) from volumes where
project_id=$(project_id) and deleted=False and volume_type=$(volume_type)

      2. *Quota Reserve/Quota Commit* stands for total time consumed when
executing QUOTA.reserve and QUOTA.commit.

1. Create 1000 volumes in tenant which has *10000* records in database
and *180000
*undeleted records in total:
[image: image.png]
2.  Create 1000 volumes in tenant which has *20000* records in
database and *180000
*undeleted records in total:

[image: image.png]
3.  Create 1000 volumes in tenant which has *30000* records in
database and *180000
*undeleted records in total:
[image: image.png]
4.  Create 1000 volumes in tenant which has* 40000* records in
database and *180000
*undeleted records in total:

[image: image.png]
5.  Create 1000 volumes in tenant which has* 60000* records in
database and *180000
*undeleted records in total:
[image: image.png]
I only posted some of the test results here, but in general, the new system
will become slower when the amount of concurrency or existing volumes in
tenant keeps raising.  Also it seems our current quota system will always
beat the new one in performance when there are about *30000 *volumes in
tenant.

I am a little worried about the performance impact if we replace our
current design with count resource feature, and I could be wrong, maybe I
missed something important during testing, please let me know if you have
any idea or suggestion.

Thanks
TommyLike

[1]: https://review.openstack.org/#/c/536341/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180309/65640f1b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 138510 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180309/65640f1b/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 113838 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180309/65640f1b/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 116220 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180309/65640f1b/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 100314 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180309/65640f1b/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 90341 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180309/65640f1b/attachment-0009.png>