Open Stack

Thanks Gorka,
    To be clear, I started this discussion not because I reject this
feature, instead I like it as it's much more clean and simple, compared
with performance impact it solves several other issues which we hate badly.
I wrote this is to point out we may have this issue, and to see whether we
could improve it before it's actually landed. Better is better:)



*    - Using a single query to retrieve both counts and sums instead of 2
   queries.*

For this advice, I think I already combined count and sum into single query.

 - DB triggers to do the actual counting.

This seems a good idea, but not sure whether it could cover all of the
cases we have in our quota system and whether can be easily integrated into
cinder, can you share more detail on this?

Thanks
TommyLike


Gorka Eguileor <geguileo at redhat.com>于2018年3月9日周五 下午8:11写道：

> On 09/03, TommyLike Hu wrote:
> > Hey all,
> >     During the Cinder and Manila's cross project discussion on quota
> system
> > last week, I raise our concern of performance impact on the count
> resource
> > feature, and Gorka pointed out we might miss some of the indexes when
> > testing. So I reshaped the environment and share some test results here
> as
> > below:
> >
> > *Test patch:*
> >      Basically this [1] is borrowed from Nova for Cinder, and only the
> > create volume process has been updated to use the new system.
> >
> > *Test Environment on AWS:*
> >     1. 3 EC2 t2.large(2 vCPU, 8GiB，80G SSD) each one deployed cinder API
> > service with 10 api workers (coordinated by haproxy)
> >     2. 1 RDS db.m4.xlarge(4 vCPU, 16GiB，200G SSD) MySQL 5.7.19
> >
> > *Database upgrade:*
> > Composite index has been added to volume table
> (volumes__composite_index):
> >
> +---------+------------+--------------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
> > | Table   | Non_unique | Key_name                      | Seq_in_index |
> > Column_name         | Collation | Cardinality | Sub_part | Packed | Null
> |
> > Index_type | Comment | Index_comment |
> >
> +---------+------------+--------------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
> > |   .....other index.......     |
> > | volumes |                 1 | volumes__composite_index |             1
> |
> > project_id          | A         |           2 |     NULL | NULL   | YES
> |
> > BTREE      |         |               |
> > | volumes |                 1 | volumes__compoite_index |
>  2 |
> > deleted             | A         |           2 |     NULL | NULL   | YES
> |
> > BTREE      |         |               |
> > | volumes |                 1 | volumes__composite_index |             3
> |
> > volume_type_id      | A         |           2 |     NULL | NULL   | YES
> |
> > BTREE      |         |               |
> >
> +---------+------------+--------------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
> >  Explain result for one of the sql statements for new quota system
> (*explain
> > select count(*), sum(size) from volumes where project_id={project_id} and
> > volume_type_id={type_id} and deleted=false*):
> >
> >
> +----+-------------+---------+------------+------+--------------------------+--------------------------+---------+-------------------+------+----------+-------+
> > | id | select_type | table   | partitions | type | possible_keys
> >  | key                      | key_len | ref               | rows |
> filtered
> > | Extra |
> >
> >
> +----+-------------+---------+------------+------+--------------------------+--------------------------+---------+-------------------+------+----------+-------+
> > |  1 | SIMPLE      | volumes | NULL       | ref  |
> volumes__composite_index
> > | volumes__composite_index | 881     | const,const,const |    1 |
>  100.00
> > | NULL  |
> >
> >
> +----+-------------+---------+------------+------+--------------------------+--------------------------+---------+-------------------+------+----------+-------+
> >
> > *Time comparsion between two system (in Seconds, * *Conc =
> Concurrency**):*
> >
> > *NOTE**:*
> >
> >       1. *QUOTAS.check_deltas*  stands for the total time consumed
> > including two sql statements as below when creating single volume:
> >
> >           1. SELECT count(id), sum(size) from volumes where
> > project_id=$(project_id) and deleted=False
> >
> >           2. SELECT count(id), sum(size) from volumes where
> > project_id=$(project_id) and deleted=False and volume_type=$(volume_type)
> >
>
> Hi,
>
> As I see it there are 3 questions here:
>
> 1. Do we need to change the quota system?
>
>    I believe we all agree that the Quota system in Cinder is bad, so bad
>    that we can divide Cinder deployments in 2 clear categories, those
>    that don't use quotas and those that have out of sync quotas.
>
> 2. Will the DB implementation used by Nova solve our current problems?
>
>    As the Nova team kindly explained us, the new quotas may not be
>    perfect for limiting (we may allow going slightly above allowed quota
>    or just get short), but it will at least solve our problems of having
>    out of sync quotas that require manually changing the DB to fix it.
>
> 3. Will the new solution introduce new problems?
>
>    To me introducing a small performance impact on resource creation is
>    an acceptable trade-off compared with the alternative, moreover
>    considering that most resource creation procedures are somewhat slow
>    operations.
>
>    And let's not forget that we can always look for more efficient ways
>    of doing the counting:
>
>    - Using a single query to retrieve both counts and sums instead of 2
>      queries.
>
>    - DB triggers to do the actual counting.
>
> To me comparing the performance of something that doesn't work with
> something that does doesn't seem fair.
>
> Cheers,
> Gorka.
>
>
>
> >       2. *Quota Reserve/Quota Commit* stands for total time consumed when
> > executing QUOTA.reserve and QUOTA.commit.
> >
> > 1. Create 1000 volumes in tenant which has *10000* records in database
> > and *180000
> > *undeleted records in total:
> > [image: image.png]
> > 2.  Create 1000 volumes in tenant which has *20000* records in
> > database and *180000
> > *undeleted records in total:
> >
> > [image: image.png]
> > 3.  Create 1000 volumes in tenant which has *30000* records in
> > database and *180000
> > *undeleted records in total:
> > [image: image.png]
> > 4.  Create 1000 volumes in tenant which has* 40000* records in
> > database and *180000
> > *undeleted records in total:
> >
> > [image: image.png]
> > 5.  Create 1000 volumes in tenant which has* 60000* records in
> > database and *180000
> > *undeleted records in total:
> > [image: image.png]
> > I only posted some of the test results here, but in general, the new
> system
> > will become slower when the amount of concurrency or existing volumes in
> > tenant keeps raising.  Also it seems our current quota system will always
> > beat the new one in performance when there are about *30000 *volumes in
> > tenant.
> >
> > I am a little worried about the performance impact if we replace our
> > current design with count resource feature, and I could be wrong, maybe I
> > missed something important during testing, please let me know if you have
> > any idea or suggestion.
> >
> > Thanks
> > TommyLike
> >
> > [1]: https://review.openstack.org/#/c/536341/
>
>
>
>
>
>
>
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180309/9868403c/attachment.html>