[openstack-dev] [cross-project][quotas][delimiter]My thoughts on how Delimiter uses generation-id for sequencing
Amrith Kumar
amrith at tesora.com
Sun May 15 17:00:33 UTC 2016
I'm not thrilled that there are two API's one for a quota check and one
for a consumption, and where it is up to the caller to properly return
the results of one to the other.
I'd much rather have consume accept the quotas and the request and 'just
do it'. With that approach, the generation id's and things are entirely
out of the requesters reach.
I'll send a simple example illustrating the use of generations (which I
hope will be simpler than Jay's example below).
-amrith
On Sun, 2016-05-15 at 11:06 -0400, Jay Pipes wrote:
> On 05/15/2016 04:16 AM, Qijing Li wrote:
> > Hi Vilobh,
> >
> > Here is my thoughts on how Delimiter uses generation-id to guarantee
> > sequencing. Please correct me if I understand it wrong.
> >
> > First, the Delimiter need to introduce another model ResourceProvider
> > who has two attributes:
> >
> > * resource_id
> > * generation_id
>
> We will need generations for *consumers* of resources as well. The
> generation for a provider of a resource is used when updating that
> particular provider's view of its inventory. For quotas to work
> effectively, we need each service to keep a generation for each consumer
> of resources in the system.
>
> > The followings are the steps of how to consume a quota:
> >
> > Step 1. Check if there is enough available quota
>
> When you refer to quota above, you are using incorrect terminology. The
> quota is the *limit* that a consumer of a resource has for a particular
> class of resources. There is no such thing as "available quota". What
> you are referring to above is whether the requested amount of resources
> for a particular consumer would exceed that consumer's quota for a resource.
>
> > If yes, then get the $generation_id //by querying the model
> > ResourceProvider with the given resource_id which is the point in time
> > view of resource usage.
>
> The generation for a resource provider is not used when *checking* if
> quota would be exceeded for a consumer's request of a particular
> resource class. The generation for a resource provider is used when
> *consuming* resources on a particular resource provider. This
> consumption process doesn't have anything to do with Delimiter, though.
> It is an internal mechanism of each service whether it uses heavy
> locking techniques or whether it uses a generation and retries to ensure
> a consistent view.
>
> Please see example code below.
>
> > If no, terminate the process of consuming the quota and return the
> > message of “No enough quotas available."
>
> Yes.
>
> > Step 2. Consume the quota.
> >
> > 2.1 Begin transaction
> >
> > 2.2 Update the QuotaUsage model: QuotaUsage.in_use =
> > QuotaUsage.in_use + amount of quota requested.
>
> No. The above is precisely why there are lots of problems in the
> existing system. The QuotaUsage model and database tables need to go
> away entirely. They represent a synchronicity problem because they
> contain duplicate data (the amount/sum used) from the actual resource
> usage tables in the services.
>
> Delimiter should not be responsible for updating any service's view of
> resource usage. That is the responsibility of the service itself to do
> this. All Delimiter needs to do is supply an interface/object model by
> which services should represent usage records and an interface by which
> services can determine if a consumer has concurrently changed its
> consumption of resources in that service.
>
> > 2.3 Get the $generation_id by querying the ResourceProvider by the
> > given resource_id.
> >
> > If the $generation_id is larger than the $generation_id in Step
> > 1, then roll back transaction and GOTO step 1.
> >
> > this case means there is someone else has changed the
> > QuotaUsage during this process.
> >
> > If the $generation_id is the same as the $generation_id in Step
> > 1, then increase the ResourceProvider.generation_id by one and
> >
> > Commit the transaction. Done!
> >
> > Note: no case the $generation_id is less than the
> > $generation_id in Step 1 because the $generation_id is nondecreasing.
>
> No, sorry, the code in my earlier response to Vilobh and Nikhil was
> confusing. The consumer's generation is what needs to be supplied by
> Delimiter. The resource provider's generation is used by the service
> itself to ensure a consistent view of usages across multiple concurrent
> consumers. The resource provider's generation is an internal mechanism
> the service could use to prevent multiple consumers from exceeding the
> provider's available resources.
>
> Here is what I think needs to be the "interface" that Delimiter facilitates:
>
> ```python
> import sqlalchemy as sa
> from sqlalchemy import sql
>
> import delimiter
> from delimiter import objects as d_objects
> from nova import objects as n_objects
> from nova.db.sqlalchey import tables as n_tables
>
>
> class NoRowsMatched(Exception):
> pass
>
>
> class ConcurrentConsumption(Exception)
> pass
>
>
> def nova_check(quotas, request_spec):
> """
> Do a verification that the resources requested by the supplied user
> and tenant involved in the request specification do not cause the
> user or tenant's quotas to be exceeded.
>
> :param request_spec: `delimiter.objects.RequestSpec` object
> containing requested resource amounts, the
> requesting user and project, etc.
> :returns `delimiter.objects.QuotaCheckResult` object
> containing the boolean result of the check, the
> resource classes that violated quotas, and a generation
> for the user.
> """
> res = d_objects.QuotaCheckResult()
> alloc_tbl = n_tables.ALLOCATIONS
> cons_tbl = n_tables.CONSUMERS
> req_resources = request_spec.resources
>
> conn = get_sql_connection(...)
>
> # Grab our consumer's generation ID
> query = sa.select([cons_tbl.c.generation])
> query = query.select_from(cons_tbl)
> query = query.where(cons_tbl.c.id == request_spec.user_id)
> cons_gen = conn.execute(query).fetchone()[0]
> res.consumer_generation = cons_gen
>
> # Grab usage amounts
> cols = [
> alloc_tbl.c.resource_class_id,
> sql.func.sum(alloc_tbl.c.used).label('total_used'),
> ]
> where_conds = [
> alloc_tbl.c.consumer_id = request_spec.user_id,
> alloc_tbl.c.resource_class_id.in_(
> req_resources.keys()
> ),
> ]
> query = sa.select(cols).select_from(alloc_tbl)
> query = query.where(where_conds)
> query = query.group_by(alloc_tbl.c.resource_class_id)
>
> records = conn.execute(query)
> usage_map = {r[0]: r[1] for r in records}
>
> res.success = True
> # Determine if any quotas are exceeded
> for resource_class, req_amount in req_resources.items():
> limit = quotas.get(resource_class)
> used = usage_map.get(resource_class)
> if limit is None or used is None:
> continue
> if (req_amount + used) > limit:
> res.success &= False
> res.exceeded.append(resource_class)
>
> return res
>
>
> def nova_consume(check_result, request_spec):
> """
> Actually writes allocation records to the database and
> increments the supplied consumer's generation to ensure
> a consistent view of usage.
>
> :param check_result: `delimiter.objects.QuotaCheckResult` object
> returned from `delimiter.check()` call
> :param request_spec: `delimiter.objects.RequestSpec` object
> containing requested resource amounts, the
> requesting user and project, etc.
> :raises `ConcurrentConsumption` if the final increment of
> the consumer's generation failed to affect a single row
> :raises `NoValidHost` if no resource provider could be found
> with capacity to handle the requested resources.
> """
> rp_tbl = n_tables.RESOURCE_PROVIDERS
> alloc_tbl = n_tables.ALLOCATIONS
> cons_tbl = n_tables.CONSUMERS
> conn = get_sql_connection(...)
> req_resources = request_spec.resources
>
> max_retries = 3
> tries = 1
>
> while True:
> # Nova finds a resource provider that has capacity for the
> # requested set of resources. The objects returned by the call
> # to find_resource_provider() are of type
> # `nova.objects.ResourceProvider`. These objects have a
> # generation property that allows Nova to have a consistent view
> # of inventory on each individual resource provider.
> provider = find_resource_provider(request_spec)
>
> if provider is None:
> raise NoValidHost
>
> trans = conn.begin()
> try:
> # Write allocation records for all requested resources
> for resource_class, req_amount in req_resources.items():
> ins = alloc_tbl.insert().values(
> resource_provider_id=provider.id,
> resource_class_id=resource_class,
> consumer_id=request_spec.user_id,
> used=req_amount)
> conn.execute(ins)
>
> # Update the generation of the resource provider
> generation = provider.generation
> where_conds = [
> rp_tbl.c.id == provider_id,
> rp_tbl.c.generation == generation,
> ]
> upd = rp_tbl.update().where(sql.and_(*where_conds))
> upd = upd.values(generation=generation+1)
> upd_res = conn.execute(upd)
>
> # Check to see if another thread executed a concurrent
> # update on the same target resource provider
> rc = upd_res.rowcount
> if rc != 1:
> raise NoRowsMatched
>
> # Update the generation of the consumer
> generation = check_res.consumer_generation
> where_conds = [
> cons_tbl.c.id == request_spec.user_id,
> cons_tbl.c.generation == generation,
> ]
> upd = cons_tbl.update().where(sql.and_(*where_conds))
> upd = upd.values(generation=generation+1)
> upd_res = conn.execute(upd)
>
> # Check to see if another client of the same consumer
> # consumed some resources in between the quota check
> # and here...
> rc = upd_res.rowcount
> if rc != 1:
> raise ConcurrentConsumption()
>
> trans.commit()
>
> except ConcurrentConsumption:
> # This exception triggers the outer loop to recheck
> # quota is not violated and attempt to consume again
> trans.rollback()
> raise
> except NoRowsMatched:
> trans.rollback()
> if tries >= max_retries:
> raise NoValidHost
> tries += 1
>
>
> # Initialize Delimiter library interfaces to use the Nova service's
> # callbacks defined above for checking and resource consumption
> delimiter.set_check_cb(nova_check)
> delimiter.set_consume_cb(nova_consume)
>
> # Assume a user 'uX' in project 'pA' requests 2 VCPU resources
> # and 1024 MEMORY_MB resources in their request to Nova:
>
> rq = d_objects.RequestSpec()
> rq.user_id = 'uX'
> rq.project_id = 'pA'
> rq.resources[VCPU] = 2
> rq.resources[MEMORY_MB] = 1024
>
> # Assume delimiter.get_quotas_for_user() queries some data store
> # for the limits for a user. The returned object is a dict of
> # limit values, keyed by resource class ID.
> quotas = delimiter.get_quotas_for_user(rq.user_id)
>
> # We first do a check of user X and the requested resource amount
> check_res = delimiter.check(quotas, rq)
>
> # Just exit if we would have violated any quotas
> if not check_res.success:
> raise delimiter.QuotaExceeded(resources=check_res.exceeded)
>
> # Now do the consumption, allowing the service itself to use
> # the consumer's generation ID to ensure a consistent view
>
> max_retries = 3
> tries = 1
> while True:
> try:
> delimiter.consume(check_res, rq)
> except NoValidHost:
> # Nothing we can do about this... just raise.
> raise
> except ConcurrentConsumption:
> # Another thread consumed for this user in between our first
> # check and our consume() call. Let's recheck and try to
> # consume again.
> if tries >= max_retries:
> raise RetriesExceeded
> check_res = delimiter.check(quotas, rq)
> if not check_res.success:
> raise delimiter.QuotaExceeded(resources=check_res.exceeded)
> tries += 1
> ```
>
> The above code demonstrates how Nova would write the `check` and
> `consume` callbacks to ensure that its view of resource provider
> inventory and usage, as well as its view of a consumer's consumption of
> any resource in the system is kept consistent.
>
> Other services could certainly use the same compare-and-update technique
> that the nova_consume() method uses above, but they are not required to.
> They could alternatively use a heavier lock-based system to ensure
> consistency or they could use a DLM of some sorts, or they could even
> forgo locking entirely and build some level of tolerance for exceeding
> quotas into their system and rely on post-consume auditing or the like.
>
> The point of the above is to show my idea of where I think Delimiter's
> scope should be and where I believe the services themselves need to own
> certain pieces of the request-check-consume workflow.
>
> Best,
> -jay
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 966 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160515/29eec74f/attachment.pgp>
More information about the OpenStack-dev
mailing list