Open Stack

On 05/15/2016 04:16 AM, Qijing Li wrote:
> Hi Vilobh,
>
> Here is my thoughts on how Delimiter uses generation-id to guarantee
>   sequencing. Please correct me if I understand it wrong.
>
> First, the Delimiter need to introduce another model ResourceProvider
> who has two attributes:
>
>   * resource_id
>   * generation_id

We will need generations for *consumers* of resources as well. The 
generation for a provider of a resource is used when updating that 
particular provider's view of its inventory. For quotas to work 
effectively, we need each service to keep a generation for each consumer 
of resources in the system.

> The followings are the steps of how to consume a quota:
>
> Step 1. Check if there is enough available quota

When you refer to quota above, you are using incorrect terminology. The 
quota is the *limit* that a consumer of a resource has for a particular 
class of resources. There is no such thing as "available quota". What 
you are referring to above is whether the requested amount of resources 
for a particular consumer would exceed that consumer's quota for a resource.

>      If yes, then get the $generation_id //by querying the model
> ResourceProvider with the given resource_id which is the point in time
> view of resource usage.

The generation for a resource provider is not used when *checking* if 
quota would be exceeded for a consumer's request of a particular 
resource class. The generation for a resource provider is used when 
*consuming* resources on a particular resource provider. This 
consumption process doesn't have anything to do with Delimiter, though. 
It is an internal mechanism of each service whether it uses heavy 
locking techniques or whether it uses a generation and retries to ensure 
a consistent view.

Please see example code below.

>      If no, terminate the process of consuming the quota and return the
> message of “No enough quotas available."

Yes.

> Step 2. Consume the quota.
>
>     2.1 Begin transaction
>
>     2.2 Update the QuotaUsage model: QuotaUsage.in_use =
> QuotaUsage.in_use + amount of quota requested.

No. The above is precisely why there are lots of problems in the 
existing system. The QuotaUsage model and database tables need to go 
away entirely. They represent a synchronicity problem because they 
contain duplicate data (the amount/sum used) from the actual resource 
usage tables in the services.

Delimiter should not be responsible for updating any service's view of 
resource usage. That is the responsibility of the service itself to do 
this. All Delimiter needs to do is supply an interface/object model by 
which services should represent usage records and an interface by which 
services can determine if a consumer has concurrently changed its 
consumption of resources in that service.

>     2.3 Get the $generation_id by querying the ResourceProvider by the
> given resource_id.
>
>          If the $generation_id is larger than the $generation_id in Step
> 1, then roll back transaction and GOTO step 1.
>
>             this case means there is someone else has changed the
> QuotaUsage during this process.
>
>          If the $generation_id is the same as the $generation_id in Step
> 1, then increase the ResourceProvider.generation_id by one and
>
>          Commit the transaction. Done!
>
>          Note: no case the $generation_id is less than the
> $generation_id in Step 1 because the $generation_id is nondecreasing.

No, sorry, the code in my earlier response to Vilobh and Nikhil was 
confusing. The consumer's generation is what needs to be supplied by 
Delimiter. The resource provider's generation is used by the service 
itself to ensure a consistent view of usages across multiple concurrent 
consumers. The resource provider's generation is an internal mechanism 
the service could use to prevent multiple consumers from exceeding the 
provider's available resources.

Here is what I think needs to be the "interface" that Delimiter facilitates:

```python
import sqlalchemy as sa
from sqlalchemy import sql

import delimiter
from delimiter import objects as d_objects
from nova import objects as n_objects
from nova.db.sqlalchey import tables as n_tables


class NoRowsMatched(Exception):
     pass


class ConcurrentConsumption(Exception)
     pass


def nova_check(quotas, request_spec):
     """
     Do a verification that the resources requested by the supplied user
     and tenant involved in the request specification do not cause the
     user or tenant's quotas to be exceeded.

     :param request_spec: `delimiter.objects.RequestSpec` object
                          containing requested resource amounts, the
                          requesting user and project, etc.
     :returns `delimiter.objects.QuotaCheckResult` object
              containing the boolean result of the check, the
              resource classes that violated quotas, and a generation
              for the user.
     """
     res = d_objects.QuotaCheckResult()
     alloc_tbl = n_tables.ALLOCATIONS
     cons_tbl = n_tables.CONSUMERS
     req_resources = request_spec.resources

     conn = get_sql_connection(...)

     # Grab our consumer's generation ID
     query = sa.select([cons_tbl.c.generation])
     query = query.select_from(cons_tbl)
     query = query.where(cons_tbl.c.id == request_spec.user_id)
     cons_gen = conn.execute(query).fetchone()[0]
     res.consumer_generation = cons_gen

     # Grab usage amounts
     cols = [
         alloc_tbl.c.resource_class_id,
         sql.func.sum(alloc_tbl.c.used).label('total_used'),
     ]
     where_conds = [
         alloc_tbl.c.consumer_id = request_spec.user_id,
         alloc_tbl.c.resource_class_id.in_(
             req_resources.keys()
         ),
     ]
     query = sa.select(cols).select_from(alloc_tbl)
     query = query.where(where_conds)
     query = query.group_by(alloc_tbl.c.resource_class_id)

     records = conn.execute(query)
     usage_map = {r[0]: r[1] for r in records}

     res.success = True
     # Determine if any quotas are exceeded
     for resource_class, req_amount in req_resources.items():
         limit = quotas.get(resource_class)
         used = usage_map.get(resource_class)
         if limit is None or used is None:
             continue
         if (req_amount + used) > limit:
             res.success &= False
             res.exceeded.append(resource_class)

     return res


def nova_consume(check_result, request_spec):
     """
     Actually writes allocation records to the database and
     increments the supplied consumer's generation to ensure
     a consistent view of usage.

     :param check_result: `delimiter.objects.QuotaCheckResult` object
                          returned from `delimiter.check()` call
     :param request_spec: `delimiter.objects.RequestSpec` object
                          containing requested resource amounts, the
                          requesting user and project, etc.
     :raises `ConcurrentConsumption` if the final increment of
             the consumer's generation failed to affect a single row
     :raises `NoValidHost` if no resource provider could be found
             with capacity to handle the requested resources.
     """
     rp_tbl = n_tables.RESOURCE_PROVIDERS
     alloc_tbl = n_tables.ALLOCATIONS
     cons_tbl = n_tables.CONSUMERS
     conn = get_sql_connection(...)
     req_resources = request_spec.resources

     max_retries = 3
     tries = 1

     while True:
         # Nova finds a resource provider that has capacity for the
         # requested set of resources. The objects returned by the call
         # to find_resource_provider() are of type
         # `nova.objects.ResourceProvider`. These objects have a
         # generation property that allows Nova to have a consistent view
         # of inventory on each individual resource provider.
         provider = find_resource_provider(request_spec)

         if provider is None:
             raise NoValidHost

         trans = conn.begin()
         try:
             # Write allocation records for all requested resources
             for resource_class, req_amount in req_resources.items():
                 ins = alloc_tbl.insert().values(
                      resource_provider_id=provider.id,
                      resource_class_id=resource_class,
                      consumer_id=request_spec.user_id,
                      used=req_amount)
                 conn.execute(ins)

             # Update the generation of the resource provider
             generation = provider.generation
             where_conds = [
                 rp_tbl.c.id == provider_id,
                 rp_tbl.c.generation == generation,
             ]
             upd = rp_tbl.update().where(sql.and_(*where_conds))
             upd = upd.values(generation=generation+1)
             upd_res = conn.execute(upd)

             # Check to see if another thread executed a concurrent
             # update on the same target resource provider
             rc = upd_res.rowcount
             if rc != 1:
                 raise NoRowsMatched

             # Update the generation of the consumer
             generation = check_res.consumer_generation
             where_conds = [
                 cons_tbl.c.id == request_spec.user_id,
                 cons_tbl.c.generation == generation,
             ]
             upd = cons_tbl.update().where(sql.and_(*where_conds))
             upd = upd.values(generation=generation+1)
             upd_res = conn.execute(upd)

             # Check to see if another client of the same consumer
             # consumed some resources in between the quota check
             # and here...
             rc = upd_res.rowcount
             if rc != 1:
                 raise ConcurrentConsumption()

             trans.commit()

         except ConcurrentConsumption:
             # This exception triggers the outer loop to recheck
             # quota is not violated and attempt to consume again
             trans.rollback()
             raise
         except NoRowsMatched:
             trans.rollback()
             if tries >= max_retries:
                 raise NoValidHost
             tries += 1


# Initialize Delimiter library interfaces to use the Nova service's
# callbacks defined above for checking and resource consumption
delimiter.set_check_cb(nova_check)
delimiter.set_consume_cb(nova_consume)

# Assume a user 'uX' in project 'pA' requests 2 VCPU resources
# and 1024 MEMORY_MB resources in their request to Nova:

rq = d_objects.RequestSpec()
rq.user_id = 'uX'
rq.project_id = 'pA'
rq.resources[VCPU] = 2
rq.resources[MEMORY_MB] = 1024

# Assume delimiter.get_quotas_for_user() queries some data store
# for the limits for a user. The returned object is a dict of
# limit values, keyed by resource class ID.
quotas = delimiter.get_quotas_for_user(rq.user_id)

# We first do a check of user X and the requested resource amount
check_res = delimiter.check(quotas, rq)

# Just exit if we would have violated any quotas
if not check_res.success:
     raise delimiter.QuotaExceeded(resources=check_res.exceeded)

# Now do the consumption, allowing the service itself to use
# the consumer's generation ID to ensure a consistent view

max_retries = 3
tries = 1
while True:
     try:
         delimiter.consume(check_res, rq)
     except NoValidHost:
         # Nothing we can do about this... just raise.
         raise
     except ConcurrentConsumption:
         # Another thread consumed for this user in between our first
         # check and our consume() call. Let's recheck and try to
         # consume again.
         if tries >= max_retries:
             raise RetriesExceeded
         check_res = delimiter.check(quotas, rq)
         if not check_res.success:
             raise delimiter.QuotaExceeded(resources=check_res.exceeded)
         tries += 1
```

The above code demonstrates how Nova would write the `check` and 
`consume` callbacks to ensure that its view of resource provider 
inventory and usage, as well as its view of a consumer's consumption of 
any resource in the system is kept consistent.

Other services could certainly use the same compare-and-update technique 
that the nova_consume() method uses above, but they are not required to. 
They could alternatively use a heavier lock-based system to ensure 
consistency or they could use a DLM of some sorts, or they could even 
forgo locking entirely and build some level of tolerance for exceeding 
quotas into their system and rely on post-consume auditing or the like.

The point of the above is to show my idea of where I think Delimiter's 
scope should be and where I believe the services themselves need to own 
certain pieces of the request-check-consume workflow.

Best,
-jay