[openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

Jay Pipes jaypipes at gmail.com
Tue Apr 26 16:33:27 UTC 2016


On 04/25/2016 02:05 PM, Joshua Harlow wrote:
> Is the generation stuff going to be exposed outside of the AP?

No, wasn't planning on it.

> I'm sort of hoping not(?), because a service (if one everyone wanted to
> create it) for say zookeeper (or etcd or consul...) could use its
> built-in generation equivalent (every znode has a version that u can use
> to do equivalent things).
>
> Thus it's like the gist I posted earlier @
>
> https://gist.github.com/harlowja/e7175c2d76e020a82ae94467a1441d85
>
> So might be nice to not expose such a thing outside of the db-layer.
>
> Amrith Kumar wrote:
>> On Sat, 2016-04-23 at 21:41 +0000, Amrith Kumar wrote:
>>> Ok to beer and high bandwidth. FYI Jay the distributed high perf db we
>>> did a couple of years ago is now open source. Just saying. Mysql plug
>>> compatible ....
>>
>>> -amrith
>>>
>>>
>>> --
>>> Amrith Kumar
>>> amrith at tesora.com
>>>
>>>
>>> -------- Original message --------
>>> From: Jay Pipes<jaypipes at gmail.com>
>>> Date: 04/23/2016 4:10 PM (GMT-05:00)
>>> To: Amrith Kumar<amrith at tesora.com>,
>>> openstack-dev at lists.openstack.org
>>> Cc: vilobhmm at yahoo-inc.com, nik.komawar at gmail.com, Ed Leafe
>>> <ed at leafe.com>
>>> Subject: Re: [openstack-dev] More on the topic of DELIMITER, the Quota
>>> Management Library proposal
>>>
>>>
>>> Looking forward to arriving in Austin so that I can buy you a beer,
>>> Amrith, and have a high-bandwidth conversation about how you're
>>> wrong. :P
>>
>>
>> Jay and I chatted and it took a long time to come to an agreement
>> because we weren't able to find any beer.
>>
>> Here's what I think we've agreed about. The library will store data in
>> two tables;
>>
>> 1. the detail table which stores the individual claims and the resource
>> class.
>> 2. a generation table which stores the resource class and a generation.
>>
>> When a claim is received, the requestor performs the following
>> operations.
>>
>> begin
>>
>>          select sum(detail.claims) as total_claims,
>>              generation.resource as resource,
>>              generation.generation last_generation
>>          from detail, generation
>>          where detail.resource = generation.resource
>>          and generation.resource =<chosen resource; memory, cpu, ...>
>>          group by generation.generation, generation.resource
>>
>>          if total_claims + this_claim<  limit
>>                  insert into detail values (this_claim, resource)
>>
>>                  update generation
>>                  set generation = generation + 1
>>                  where generation = last_generation
>>
>>                  if @@rowcount = 1
>>                          -- all good
>>                          commit
>>                  else
>>                          rollback
>>                          -- try again
>>
>>
>>
>> There will be some bootstrapping that will be required for the situation
>> where there are no detail records for a given resource and so on but I
>> think we can figure that out easily. The easiest way I can think of
>> doing that is to lose the join and do both the queries (one against the
>> detail and one against the generation table within the same
>> transaction).
>>
>> Using the generation table update as the locking mechanism that prevents
>> multiple requestors from making concurrent claims.
>>
>> So long as people don't go and try and read these tables and change
>> tables outside of the methods that the library provides, we can
>> guarantee that this is al safe and will not oversubscribe.
>>
>> -amrith
>>
>>> Comments inline.
>>>
>>> On 04/23/2016 11:25 AM, Amrith Kumar wrote:
>>>> On Sat, 2016-04-23 at 10:26 -0400, Andrew Laski wrote:
>>>>> On Fri, Apr 22, 2016, at 09:57 PM, Tim Bell wrote:
>>>>>
>>>>>> I have reservations on f and g.
>>>>>>
>>>>>>
>>>>>> On f., We have had a number of discussions in the past about
>>>>>> centralising quota (e.g. Boson) and the project teams of the other
>>>>>> components wanted to keep the quota contents ‘close’. This can
>>>>>> always be reviewed further with them but I would hope for at least
>>> a
>>>>>> standard schema structure of tables in each project for the
>>> handling
>>>>>> of quota.
>>>>>>
>>>>>>
>>>>>> On g., aren’t all projects now nested projects ? If we have the
>>>>>> complexity of handling nested projects sorted out in the common
>>>>>> library, is there a reason why a project would not want to support
>>>>>> nested projects ?
>>>>>>
>>>>>>
>>>>>> One other issue is how to do reconcilliation, each project needs
>>> to
>>>>>> have a mechanism to re-calculate the current allocations and
>>>>>> reconcile that with the quota usage. While in an ideal world, this
>>>>>> should not be necessary, it would be for the foreseeable future,
>>>>>> especially with a new implementation.
>>>>>>
>>>>> One of the big reasons that Jay and I have been pushing to remove
>>>>> reservations and tracking of quota in a separate place than the
>>>>> resources are actually used, e.g., an instance record in the Nova
>>> db,
>>>>> is so that reconciliation is not necessary. For example, if RAM
>>> quota
>>>>> usage is simply tracked as sum(instances.memory_mb) then you can be
>>>>> sure that usage is always up to date.
>>>> Uh oh, there be gremlins here ...
>>>>
>>>> I am positive that this will NOT work, see earlier conversations
>>> about
>>>> isolation levels, and Jay's alternate solution.
>>>>
>>>> The way (I understand the issue, and Jay's solution) you get around
>>> the
>>>> isolation levels trap is to NOT do your quota determinations based
>>> on a
>>>> SUM(column) but rather based on the rowcount on a well crafted
>>> UPDATE of
>>>> a single table that stored total quota.
>>> No, we would do our quota calculations by doing a SUM(used) against
>>> the
>>> allocations table. There is no separate table that stored the total
>>> quota (or quota usage records). That's the source of the problem with
>>> the existing quota handling code in Nova. The generation field value
>>> is
>>> used to provide the consistent view of the actual resource usage
>>> records
>>> so that the INSERT operations for all claimed resources can be done in
>>> a
>>> transactional manner and will be rolled back if any other writer
>>> changes
>>> the amount of consumed resources on a provider (which of course would
>>> affect the quota check calculations).
>>>
>>>   >  You could also store a detail
>>>> claim record for each claim in an independent table that is
>>> maintained
>>>> in the same database transaction if you so desire, that is optional.
>>> The allocations table is the "detail claim record" table that you
>>> refer
>>> to above.
>>>
>>>> My view of how this would work (which I described earlier as
>>> building on
>>>> Jay's solution) is that the claim flow would look like this:
>>>>
>>>>           select total_used, generation
>>>>           from quota_claimed
>>>>           where tenant =<tenant>  and resource = 'memory'
>>> There is no need to keep a total_used value for anything. That is
>>> denormalized calculated data that merely adds a point of race
>>> contention. The quota check is against the *detail* table
>>> (allocations),
>>> which stores the *actual resource usage records*.
>>>
>>>>           begin transaction
>>>>
>>>>           update quota_claimed
>>>>                   set total_used = total_used + claim, generation =
>>>>                   generation + 1
>>>>                   where tenant =<tenant>  and resource = 'memory'
>>>>                   and generation = generation
>>>>                   and total_used + claim<  limit
>>> This part of the transaction must always occur **after** the
>>> insertion
>>> of the actual resource records, not before.
>>>
>>>>           if @@rowcount = 1
>>>>                   -- optional claim_detail table
>>>>                   insert into claim_detail values (<tenant>,
>>> 'memory',
>>>>                   claim, ...)
>>>>                   commit
>>>>           else
>>>>                   rollback
>>> So, in pseudo-Python-SQLish code, my solution works like this:
>>>
>>> limits = get_limits_from_delimiter()
>>> requested = get_requested_from_request_spec()
>>>
>>> while True:
>>>
>>>       used := SELECT
>>>                 resource_class,
>>>                 resource_provider,
>>>                 generation,
>>>                 SUM(used) as total_used
>>>               FROM allocations
>>>               JOIN resource_providers ON (...)
>>>               WHERE consumer_uuid = $USER_UUID
>>>               GROUP BY
>>>                 resource_class,
>>>                 resource_provider,
>>>                 generation;
>>>
>>>       # Check that our requested resource amounts don't exceed quotas
>>>       if not check_requested_within_limits(requested, used, limits):
>>>           raise QuotaExceeded
>>>
>>>       # Claim all requested resources. Note that the generation
>>> retrieved
>>>       # from the above query is our consistent view marker. If the
>>> UPDATE
>>>       # below succeeds and returns != 0 rows affected, that means there
>>>       # was no other writer that changed our resource usage in between
>>>       # this thread's claiming of resources, and therefore we prevent
>>>       # any oversubscription of resources.
>>>       begin_transaction:
>>>
>>>           provider := SELECT id, generation, ... FROM
>>> resource_providers
>>>                       JOIN (...)
>>>                       WHERE (<resource_usage_filters>)
>>>
>>>           for resource in requested:
>>>               INSERT INTO allocations (
>>>                 resource_provider_id,
>>>                 resource_class_id,
>>>                 consumer_uuid,
>>>                 used
>>>               ) VALUES (
>>>                 $provider.id,
>>>                 $resource.id,
>>>                 $USER_UUID,
>>>                 $resource.amount
>>>               );
>>>
>>>          rows_affected := UPDATE resource_providers
>>>                           SET generation = generation + 1
>>>                           WHERE id = $provider.id
>>>                           AND generation =
>>> $used[$provider.id].generation;
>>>
>>>          if $rows_affected == 0:
>>>              ROLLBACK;
>>>
>>> The only reason we would need a post-claim quota check is if some of
>>> the
>>> requested resources are owned and tracked by an external-to-Nova
>>> system.
>>>
>>> BTW, note to Ed Leafe... unless your distributed data store supports
>>> transactional semantics, you can't use a distributed data store for
>>> these types of solutions. Instead, you will need to write a whole
>>> bunch
>>> of code that does post-auditing of claims and quotas and a system
>>> that
>>> accepts that oversubscription and out-of-sync quota limits and usages
>>> is
>>> a fact of life. Not to mention needing to implement JOINs in Python.
>>>
>>>> But, it is my understanding that
>>>>
>>>>           (a) if you wish to do the SUM(column) approach that you
>>> propose,
>>>>           you must have a reservation that is committed and then you
>>> must
>>>>           re-read the SUM(column) to make sure you did not
>>> over-subscribe;
>>>>           and
>>> Erm, kind of? Oversubscription is not possible in the solution I
>>> describe because the compare-and-update on the
>>> resource_providers.generation field allows for a consistent view of
>>> the
>>> resources used -- and if that view changes during the insertion of
>>> resource usage records -- the transaction containing those insertions
>>> is
>>> rolled back.
>>>
>>>>           (b) to get away from reservations you must stop using the
>>>>           SUM(column) approach and instead use a single quota_claimed
>>>>           table to determine the current quota claimed.
>>> No. This has nothing to do with reservations.
>>>
>>>> At least that's what I understand of Jay's example from earlier in
>>> this
>>>> thread.
>>>>
>>>> Let's definitely discuss this in Austin. While I don't love Jay's
>>>> solution for other reasons to do with making the quota table a
>>> hotspot
>>>> and things like that, it is a perfectly workable solution, I think.
>>> There is no quota table in my solution.
>>>
>>> If you refer to the resource_providers table (the table that has the
>>> generation field), then yes, it's a hot spot. But hot spots in the DB
>>> aren't necessarily a bad thing if you design the underlying schema
>>> properly.
>>>
>>> More in Austin.
>>>
>>> Best,
>>> -jay
>>>
>>>>>
>>>>>> Tim
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> From: Amrith Kumar<amrith at tesora.com>
>>>>>> Reply-To: "OpenStack Development Mailing List (not for usage
>>>>>> questions)"<openstack-dev at lists.openstack.org>
>>>>>> Date: Friday 22 April 2016 at 06:51
>>>>>> To: "OpenStack Development Mailing List (not for usage questions)"
>>>>>> <openstack-dev at lists.openstack.org>
>>>>>> Subject: Re: [openstack-dev] More on the topic of DELIMITER, the
>>>>>> Quota Management Library proposal
>>>>>>
>>>>>>
>>>>>>
>>>>>>           I’ve thought more about Jay’s approach to enforcing
>>> quotas
>>>>>>           and I think we can build on and around it. With that
>>>>>>           implementation as the basic quota primitive, I think we
>>> can
>>>>>>           build a quota management API that isn’t dependent on
>>>>>>           reservations. It does place some burdens on the consuming
>>>>>>           projects that I had hoped to avoid and these will cause
>>>>>>           heartburn for some (make sure that you always request
>>>>>>           resources in a consistent order and free them in a
>>>>>>           consistent order being the most obvious).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>           If it doesn’t make it harder, I would like to see if we
>>> can
>>>>>>           make the quota API take care of the ordering of requests.
>>>>>>           i.e. if the quota API is an extension of Jay’s example
>>> and
>>>>>>           accepts some data structure (dict?) with all the claims
>>> that
>>>>>>           a project wants to make for some operation, and then
>>>>>>           proceeds to make those claims for the project in the
>>>>>>           consistent order, I think it would be of some value.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>           Beyond that, I’m on board with a-g below,
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>           -amrith
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>           From: Vilobh Meshram
>>>>>>           [mailto:vilobhmeshram.openstack at gmail.com]
>>>>>>           Sent: Friday, April 22, 2016 4:08 AM
>>>>>>           To: OpenStack Development Mailing List (not for usage
>>>>>>           questions)<openstack-dev at lists.openstack.org>
>>>>>>           Subject: Re: [openstack-dev] More on the topic of
>>> DELIMITER,
>>>>>>           the Quota Management Library proposal
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>           I strongly agree with Jay on the points related to "no
>>>>>>           reservation" , keeping the interface simple and the role
>>> for
>>>>>>           Delimiter (impose limits on resource consumption and
>>> enforce
>>>>>>           quotas).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>           The point to keep user quota, tenant quotas in Keystone
>>>>>>           sounds interestring and would need support from Keystone
>>>>>>           team. We have a Cross project session planned [1] and
>>> will
>>>>>>           definitely bring that up in that session.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>           The main thought with which Delimiter was formed was to
>>>>>>           enforce resource quota in transaction safe manner and do
>>> it
>>>>>>           in a cross-project conducive manner and it still holds
>>>>>>           true. Delimiters mission is to impose limits on
>>>>>>           resource consumption and enforce quotas in transaction
>>> safe
>>>>>>           manner. Few key aspects of Delimiter are :-
>>>>>>
>>>>>>
>>>>>>
>>>>>>           a. Delimiter will be a new Library and not a Service.
>>>>>>           Details covered in spec.
>>>>>>
>>>>>>
>>>>>>           b. Delimiter's role will be to impose limits on resource
>>>>>>           consumption.
>>>>>>
>>>>>>
>>>>>>           c. Delimiter will not be responsible for rate limiting.
>>>>>>
>>>>>>
>>>>>>           d. Delimiter will not maintain data for the resources.
>>>>>>           Respective projects will take care of keeping,
>>> maintaining
>>>>>>           data for the resources and resource consumption.
>>>>>>
>>>>>>
>>>>>>           e. Delimiter will not have the concept of "reservations".
>>>>>>           Delimiter will read or update the "actual" resource
>>> tables
>>>>>>           and will not rely on the "cached" tables. At present, the
>>>>>>           quota infrastructure in Nova, Cinder and other projects
>>> have
>>>>>>           tables such as reservations, quota_usage, etc which are
>>> used
>>>>>>           as "cached tables" to track re
>>>>>>
>>>>>>
>>>>>>           f. Delimiter will fetch the information for project
>>> quota,
>>>>>>           user quota from a centralized place, say Keystone, or if
>>>>>>           that doesn't materialize will fetch default quota values
>>>>>>           from respective service. This information will be cached
>>>>>>           since it gets updated rarely but read many times.
>>>>>>
>>>>>>
>>>>>>           g. Delimiter will take into consideration whether the
>>>>>>           project is a Flat or Nested and will make the
>>> calculations
>>>>>>           of allocated, available resources. Nested means project
>>>>>>           namespace is hierarchical and Flat means project
>>> namespace
>>>>>>           is not hierarchical.
>>>>>>
>>>>>>
>>>>>>           -Vilobh
>>>>>>
>>>>>>
>>>>>>           [1]
>>> https://www.openstack.org/summit/austin-2016/summit-schedule/events/9492
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>           On Thu, Apr 21, 2016 at 11:08 PM, Joshua Harlow
>>>>>>           <harlowja at fastmail.com>  wrote:
>>>>>>
>>>>>>
>>>>>>                   Since people will be on a plane soon,
>>>>>>
>>>>>>                   I threw this together as a example of a quota
>>> engine
>>>>>>                   (the zookeeper code does even work, and yes it
>>>>>>                   provides transactional semantics due to the nice
>>>>>>                   abilities of zookeeper znode versions[1] and its
>>>>>>                   inherent consistency model, yippe).
>>>>>>
>>>>>>
>>> https://gist.github.com/harlowja/e7175c2d76e020a82ae94467a1441d85
>>>>>>                   Someone else can fill in the db quota engine with
>>> a
>>>>>>                   similar/equivalent api if they so dare, ha. Or
>>> even
>>>>>>                   feel to say the gist/api above is crap, cause
>>> that's
>>>>>>                   ok to, lol.
>>>>>>
>>>>>>                   [1]
>>>>>>
>>> https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#Data
>>> +Access
>>>>>>
>>>>>>
>>>>>>                   Amrith Kumar wrote:
>>>>>>
>>>>>>                           Inline below ... thread is too long, will
>>>>>>                           catch you in Austin.
>>>>>>
>>>>>>
>>>>>>                                   -----Original Message-----
>>>>>>                                   From: Jay Pipes
>>>>>>                                   [mailto:jaypipes at gmail.com]
>>>>>>                                   Sent: Thursday, April 21, 2016
>>> 8:08
>>>>>>                                   PM
>>>>>>                                   To:
>>>>>>                                   openstack-dev at lists.openstack.org
>>>>>>                                   Subject: Re: [openstack-dev] More
>>> on
>>>>>>                                   the topic of DELIMITER, the Quota
>>>>>>                                   Management Library proposal
>>>>>>
>>>>>>                                   Hmm, where do I start... I think
>>> I
>>>>>>                                   will just cut to the two primary
>>>>>>                                   disagreements I have. And I will
>>>>>>                                   top-post because this email is
>>> way
>>>>>>                                   too
>>>>>>                                   big.
>>>>>>
>>>>>>                                   1) On serializable isolation
>>> level.
>>>>>>                                   No, you don't need it at all to
>>>>>>                                   prevent races in claiming. Just
>>> use
>>>>>>                                   a
>>>>>>                                   compare-and-update with retries
>>>>>>                                   strategy. Proof is here:
>>>>>>
>>>>>>
>>> https://github.com/jaypipes/placement-bench/blob/master/placement.py#L97-
>>>
>>>>>>                                   L142
>>>>>>
>>>>>>                                   Works great and prevents multiple
>>>>>>                                   writers from oversubscribing any
>>>>>>                                   resource without relying on any
>>>>>>                                   particular isolation level at
>>> all.
>>>>>>                                   The `generation` field in the
>>>>>>                                   inventories table is what allows
>>>>>>                                   multiple
>>>>>>                                   writers to ensure a consistent
>>> view
>>>>>>                                   of the data without needing to
>>> rely
>>>>>>                                   on
>>>>>>                                   heavy lock-based semantics and/or
>>>>>>                                   RDBMS-specific isolation levels.
>>>>>>
>>>>>>
>>>>>>
>>>>>>                           [amrith] this works for what it is doing,
>>> we
>>>>>>                           can definitely do this. This will work at
>>>>>>                           any isolation level, yes. I didn't want
>>> to
>>>>>>                           go this route because it is going to
>>> still
>>>>>>                           require an insert into another table
>>>>>>                           recording what the actual 'thing' is that
>>> is
>>>>>>                           claiming the resource and that insert is
>>>>>>                           going to be in a different transaction
>>> and
>>>>>>                           managing those two transactions was what
>>> I
>>>>>>                           wanted to avoid. I was hoping to avoid
>>>>>>                           having two tables tracking claims, one
>>>>>>                           showing the currently claimed quota and
>>>>>>                           another holding the things that claimed
>>> that
>>>>>>                           quota. Have to think again whether that
>>> is
>>>>>>                           possible.
>>>>>>
>>>>>>                                   2) On reservations.
>>>>>>
>>>>>>                                   The reason I don't believe
>>>>>>                                   reservations are necessary to be
>>> in
>>>>>>                                   a quota
>>>>>>                                   library is because reservations
>>> add
>>>>>>                                   a concept of a time to a claim of
>>>>>>                                   some
>>>>>>                                   resource. You reserve some
>>> resource
>>>>>>                                   to be claimed at some point in
>>> the
>>>>>>                                   future and release those
>>> resources
>>>>>>                                   at a point further in time.
>>>>>>
>>>>>>                                   Quota checking doesn't look at
>>> what
>>>>>>                                   the state of some system will be
>>> at
>>>>>>                                   some point in the future. It
>>> simply
>>>>>>                                   returns whether the system *right
>>>>>>                                   now* can handle a request *right
>>>>>>                                   now* to claim a set of resources.
>>>>>>
>>>>>>                                   If you want reservation semantics
>>>>>>                                   for some resource, that's totally
>>>>>>                                   cool,
>>>>>>                                   but IMHO, a reservation service
>>>>>>                                   should live outside of the
>>> service
>>>>>>                                   that is
>>>>>>                                   actually responsible for
>>> providing
>>>>>>                                   resources to a consumer.
>>>>>>                                   Merging right-now quota checks
>>> and
>>>>>>                                   future-based reservations into
>>> the
>>>>>>                                   same
>>>>>>                                   library just complicates things
>>>>>>                                   unnecessarily IMHO.
>>>>>>
>>>>>>
>>>>>>
>>>>>>                           [amrith] extension of the above ...
>>>>>>
>>>>>>                                   3) On resizes.
>>>>>>
>>>>>>                                   Look, I recognize some users see
>>>>>>                                   some value in resizing their
>>>>>>                                   resources.
>>>>>>                                   That's fine. I personally think
>>>>>>                                   expand operations are fine, and
>>> that
>>>>>>                                   shrink operations are really the
>>>>>>                                   operations that should be
>>> prohibited
>>>>>>                                   in
>>>>>>                                   the API. But, whatever, I'm fine
>>>>>>                                   with resizing of requested
>>> resource
>>>>>>                                   amounts. My big point is if you
>>>>>>                                   don't have a separate table that
>>>>>>                                   stores
>>>>>>                                   quota_usages and instead only
>>> have a
>>>>>>                                   single table that stores the
>>> actual
>>>>>>                                   resource usage records, you don't
>>>>>>                                   have to do *any* quota check
>>>>>>                                   operations
>>>>>>                                   at all upon deletion of a
>>> resource.
>>>>>>                                   For modifying resource amounts
>>> (i.e.
>>>>>>                                   a
>>>>>>                                   resize) you merely need to change
>>>>>>                                   the calculation of requested
>>>>>>                                   resource
>>>>>>                                   amounts to account for the
>>>>>>                                   already-consumed usage amount.
>>>>>>
>>>>>>                                   Bottom line for me: I really
>>> won't
>>>>>>                                   support any proposal for a
>>> complex
>>>>>>                                   library that takes the resource
>>>>>>                                   claim process out of the hands of
>>>>>>                                   the
>>>>>>                                   services that own those
>>> resources.
>>>>>>                                   The simpler the interface of this
>>>>>>                                   library, the better.
>>>>>>
>>>>>>
>>>>>>
>>>>>>                           [amrith] my proposal would not but this
>>>>>>                           email thread has got too long. Yes,
>>> simpler
>>>>>>                           interface, will catch you in Austin.
>>>>>>
>>>>>>                                   Best,
>>>>>>                                   -jay
>>>>>>
>>>>>>                                   On 04/19/2016 09:59 PM, Amrith
>>> Kumar
>>>>>>                                   wrote:
>>>>>>
>>>>>>                                                   -----Original
>>>>>>                                                   Message-----
>>>>>>                                                   From: Jay Pipes
>>>>>>
>>> [mailto:jaypipes at gmail.com]
>>>>>>                                                   Sent: Monday,
>>> April
>>>>>>                                                   18, 2016 2:54 PM
>>>>>>                                                   To:
>>>>>>
>>> openstack-dev at lists.openstack.org
>>>>>>                                                   Subject: Re:
>>>>>>                                                   [openstack-dev]
>>> More
>>>>>>                                                   on the topic of
>>>>>>                                                   DELIMITER, the
>>>>>>                                                   Quota Management
>>>>>>                                                   Library proposal
>>>>>>
>>>>>>                                                   On 04/16/2016
>>> 05:51
>>>>>>                                                   PM, Amrith Kumar
>>>>>>                                                   wrote:
>>>>>>
>>>>>>                                                           If we
>>>>>>                                                           therefore
>>>>>>                                                           assume
>>> that
>>>>>>                                                           this will
>>> be
>>>>>>                                                           a Quota
>>>>>>
>>> Management
>>>>>>                                                           Library,
>>>>>>                                                           it is
>>> safe
>>>>>>                                                           to assume
>>>>>>                                                           that
>>> quotas
>>>>>>                                                           are going
>>> to
>>>>>>                                                           be
>>> managed
>>>>>>                                                           on a
>>>>>>
>>> per-project
>>>>>>                                                           basis,
>>> where
>>> participating projects will use this library.
>>>>>>                                                           I believe
>>>>>>                                                           that it
>>>>>>                                                           stands to
>>>>>>                                                           reason
>>> that
>>>>>>                                                           any data
>>>>>>
>>> persistence
>>>>>>                                                           will
>>>>>>                                                           have to
>>> be
>>>>>>                                                           in a
>>>>>>                                                           location
>>>>>>                                                           decided
>>> by
>>>>>>                                                           the
>>>>>>
>>> individual
>>>>>>                                                           project.
>>>>>>
>>>>>>
>>>>>>                                                   Depends on what
>>> you
>>>>>>                                                   mean by "any data
>>>>>>                                                   persistence". If
>>> you
>>>>>>                                                   are
>>>>>>                                                   referring to the
>>>>>>                                                   storage of quota
>>>>>>                                                   values (per user,
>>>>>>                                                   per tenant,
>>>>>>                                                   global, etc) I
>>> think
>>>>>>                                                   that should be
>>> done
>>>>>>                                                   by the Keystone
>>>>>>                                                   service.
>>>>>>                                                   This data is
>>>>>>                                                   essentially an
>>>>>>                                                   attribute of the
>>>>>>                                                   user or the
>>> tenant
>>>>>>                                                   or the
>>>>>>
>>>>>>
>>>>>>                                   service endpoint itself (i.e.
>>>>>>
>>>>>>
>>>>>>                                                   global defaults).
>>>>>>                                                   This data also
>>>>>>                                                   rarely changes
>>> and
>>>>>>                                                   logically belongs
>>>>>>                                                   to the service
>>> that
>>>>>>                                                   manages users,
>>>>>>                                                   tenants, and
>>> service
>>>>>>                                                   endpoints:
>>>>>>
>>>>>>
>>>>>>                                   Keystone.
>>>>>>
>>>>>>
>>>>>>                                                   If you are
>>> referring
>>>>>>                                                   to the storage of
>>>>>>                                                   resource usage
>>>>>>                                                   records, yes,
>>>>>>                                                   each service
>>> project
>>>>>>                                                   should own that
>>> data
>>>>>>                                                   (and frankly, I
>>>>>>                                                   don't see a
>>>>>>                                                   need to persist
>>> any
>>>>>>                                                   quota usage data
>>> at
>>>>>>                                                   all, as I
>>> mentioned
>>>>>>                                                   in a
>>>>>>                                                   previous reply to
>>>>>>                                                   Attila).
>>>>>>
>>>>>>
>>>>>>                                           [amrith] You make a
>>>>>>                                           distinction that I had
>>> made
>>>>>>                                           implicitly, and it is
>>>>>>                                           important to highlight
>>> it.
>>>>>>                                           Thanks for pointing it
>>> out.
>>>>>>                                           Yes, I meant
>>>>>>                                           both of the above, and as
>>>>>>                                           stipulated. Global
>>> defaults
>>>>>>                                           in keystone
>>>>>>                                           (somehow, TBD) and usage
>>>>>>                                           records, on a per-service
>>>>>>                                           basis.
>>>>>>
>>>>>>                                                           That may
>>> not
>>>>>>                                                           be a very
>>>>>>
>>> interesting
>>>>>>                                                           statement
>>>>>>                                                           but the
>>>>>>                                                           corollary
>>>>>>                                                           is, I
>>>>>>                                                           think, a
>>>>>>                                                           very
>>>>>>
>>> significant
>>> statement;
>>>>>>                                                           it cannot
>>> be
>>>>>>                                                           assumed
>>> that
>>>>>>                                                           the
>>>>>>                                                           quota
>>>>>>
>>> management
>>> information
>>>>>>                                                           for all
>>>>>>
>>> participating projects is in
>>>>>>                                                           the same
>>>>>>                                                           database.
>>>>>>
>>>>>>
>>>>>>                                                   It cannot be
>>> assumed
>>>>>>                                                   that this
>>>>>>                                                   information is
>>> even
>>>>>>                                                   in a database at
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   all...
>>>>>>
>>>>>>
>>>>>>                                           [amrith] I don't follow.
>>> If
>>>>>>                                           the service in question
>>> is
>>>>>>                                           to be scalable,
>>>>>>                                           I think it stands to
>>> reason
>>>>>>                                           that there must be some
>>>>>>                                           mechanism by which
>>>>>>                                           instances of the service
>>> can
>>>>>>                                           share usage records (as
>>> you
>>>>>>                                           refer to
>>>>>>                                           them, and I like that
>>> term).
>>>>>>                                           I think it stands to
>>> reason
>>>>>>                                           that there
>>>>>>                                           must be some database,
>>> no?
>>>>>>                                                           A
>>>>>>
>>> hypothetical
>>>>>>                                                           service
>>>>>>                                                           consuming
>>>>>>                                                           the
>>>>>>                                                           Delimiter
>>>>>>                                                           library
>>>>>>                                                           provides
>>>>>>
>>> requesters
>>>>>>                                                           with some
>>>>>>                                                           widgets,
>>> and
>>>>>>                                                           wishes to
>>>>>>                                                           track the
>>>>>>                                                           widgets
>>> that
>>>>>>                                                           it has
>>>>>>
>>> provisioned
>>>>>>                                                           both on a
>>>>>>                                                           per-user
>>>>>>                                                           basis,
>>> and
>>>>>>                                                           on the
>>>>>>                                                           whole. It
>>>>>>                                                           should
>>>>>>                                                           therefore
>>>>>>
>>> multi-tenant
>>>>>>                                                           and able
>>> to
>>>>>>                                                           track the
>>>>>>                                                           widgets
>>> on a
>>>>>>                                                           per
>>>>>>                                                           tenant
>>> basis
>>>>>>                                                           and if
>>>>>>                                                           required
>>>>>>                                                           impose
>>>>>>                                                           limits on
>>>>>>                                                           the
>>> number
>>>>>>                                                           of
>>> widgets
>>>>>>                                                           that a
>>>>>>                                                           tenant
>>> may
>>>>>>                                                           consume
>>> at a
>>>>>>                                                           time,
>>> during
>>>>>>                                                           a course
>>> of
>>>>>>                                                           a period
>>> of
>>>>>>                                                           time, and
>>> so
>>>>>>                                                           on.
>>>>>>
>>>>>>
>>>>>>                                                   No, this last
>>> part
>>>>>>                                                   is absolutely not
>>>>>>                                                   what I think
>>> quota
>>>>>>                                                   management
>>>>>>                                                   should be about.
>>>>>>
>>>>>>                                                   Rate limiting --
>>>>>>                                                   i.e. how many
>>>>>>                                                   requests a
>>>>>>                                                   particular user
>>> can
>>>>>>                                                   make of
>>>>>>                                                   an API in a given
>>>>>>                                                   period of time --
>>>>>>                                                   should *not* be
>>>>>>                                                   handled by
>>>>>>                                                   OpenStack API
>>>>>>                                                   services, IMHO.
>>> It
>>>>>>                                                   is the
>>>>>>                                                   responsibility of
>>>>>>                                                   the
>>>>>>                                                   deployer to
>>> handle
>>>>>>                                                   this using
>>>>>>                                                   off-the-shelf
>>>>>>                                                   rate-limiting
>>>>>>                                                   solutions
>>>>>>
>>>>>>
>>>>>>                                   (open source or proprietary).
>>>>>>
>>>>>>
>>>>>>                                                   Quotas should
>>> only
>>>>>>                                                   be about the hard
>>>>>>                                                   limit of
>>> different
>>>>>>                                                   types of
>>>>>>                                                   resources that a
>>>>>>                                                   user or group of
>>>>>>                                                   users can consume
>>> at
>>>>>>                                                   a given time.
>>>>>>
>>>>>>
>>>>>>                                           [amrith] OK, good point.
>>>>>>                                           Agreed as stipulated.
>>>>>>
>>>>>>
>>>>>>                                                           Such a
>>>>>>
>>> hypothetical
>>>>>>                                                           service
>>> may
>>>>>>                                                           also
>>> consume
>>>>>>                                                           resources
>>>>>>                                                           from
>>> other
>>>>>>                                                           services
>>>>>>                                                           that it
>>>>>>                                                           wishes to
>>>>>>                                                           track,
>>> and
>>>>>>                                                           impose
>>>>>>                                                           limits
>>> on.
>>>>>>
>>>>>>                                                   Yes, absolutely
>>>>>>                                                   agreed.
>>>>>>
>>>>>>
>>>>>>                                                           It is
>>> also
>>> understood
>>>>>>                                                           as Jay
>>> Pipes
>>>>>>                                                           points
>>> out
>>>>>>                                                           in [4]
>>> that
>>>>>>                                                           the
>>> actual
>>>>>>                                                           process
>>> of
>>> provisioning
>>>>>>                                                           widgets
>>>>>>                                                           could be
>>>>>>                                                           time
>>>>>>                                                           consuming
>>>>>>                                                           and it is
>>>>>>
>>> ill-advised
>>>>>>                                                           to hold a
>>>>>>                                                           database
>>>>>>
>>> transaction
>>>>>>                                                           of any
>>> kind
>>>>>>                                                           open for
>>>>>>                                                           that
>>>>>>                                                           duration
>>> of
>>>>>>                                                           time.
>>>>>>                                                           Ensuring
>>>>>>                                                           that a
>>> user
>>>>>>                                                           does not
>>>>>>                                                           exceed
>>> some
>>>>>>                                                           limit on
>>>>>>                                                           the
>>> number
>>>>>>                                                           of
>>>>>>
>>> concurrent
>>>>>>                                                           widgets
>>> that
>>>>>>                                                           he or she
>>>>>>                                                           may
>>> create
>>>>>>                                                           therefore
>>>>>>                                                           requires
>>>>>>                                                           some
>>>>>>                                                           mechanism
>>> to
>>>>>>                                                           track
>>>>>>                                                           in-flight
>>>>>>                                                           requests
>>> for
>>>>>>                                                           widgets.
>>> I
>>>>>>                                                           view
>>> these
>>>>>>                                                           as
>>> "intent"
>>>>>>                                                           but not
>>> yet
>>> materialized.
>>>>>>
>>>>>>                                                   It has nothing to
>>> do
>>>>>>                                                   with the amount
>>> of
>>>>>>                                                   concurrent
>>> widgets
>>>>>>                                                   that a
>>>>>>                                                   user can create.
>>>>>>                                                   It's just about
>>> the
>>>>>>                                                   total number of
>>> some
>>>>>>                                                   resource
>>>>>>                                                   that may be
>>> consumed
>>>>>>                                                   by that user.
>>>>>>
>>>>>>                                                   As for an
>>> "intent",
>>>>>>                                                   I don't believe
>>>>>>                                                   tracking intent
>>> is
>>>>>>                                                   the right way
>>>>>>                                                   to go at all. As
>>>>>>                                                   I've mentioned
>>>>>>                                                   before, the major
>>>>>>                                                   problem in Nova's
>>>>>>                                                   quota system is
>>> that
>>>>>>                                                   there are two
>>> tables
>>>>>>                                                   storing resource
>>>>>>                                                   usage
>>>>>>                                                   records: the
>>>>>>                                                   *actual* resource
>>>>>>                                                   usage tables (the
>>>>>>                                                   allocations table
>>> in
>>>>>>                                                   the new
>>>>>>                                                   resource-
>>> providers
>>>>>>                                                   modeling and the
>>>>>>                                                   instance_extra,
>>>>>>                                                   pci_devices and
>>>>>>                                                   instances table
>>> in
>>>>>>                                                   the legacy
>>> modeling)
>>>>>>                                                   and the *quota
>>>>>>                                                   usage* tables
>>>>>>                                                   (quota_usages and
>>>>>>                                                   reservations
>>>>>>                                                   tables). The
>>>>>>                                                   quota_usages
>>> table
>>>>>>                                                   does
>>>>>>                                                   not need to exist
>>> at
>>>>>>                                                   all, and neither
>>>>>>                                                   does the
>>>>>>                                                   reservations
>>> table.
>>>>>>                                                   Don't do
>>>>>>                                                   intent-based
>>>>>>                                                   consumption.
>>>>>>                                                   Instead, just
>>>>>>                                                   consume (claim)
>>> by
>>>>>>                                                   writing a record
>>> for
>>>>>>                                                   the resource
>>> class
>>>>>>                                                   consumed on a
>>>>>>                                                   provider into
>>>>>>                                                   the actual
>>> resource
>>>>>>                                                   usages table and
>>>>>>                                                   then "check
>>> quotas"
>>>>>>                                                   by querying
>>>>>>                                                   the *actual*
>>>>>>                                                   resource usages
>>> and
>>>>>>                                                   comparing the
>>>>>>                                                   SUM(used) values,
>>>>>>                                                   grouped by
>>> resource
>>>>>>                                                   class, against
>>> the
>>>>>>                                                   appropriate quota
>>>>>>                                                   limits for
>>>>>>                                                   the user. The
>>>>>>                                                   introduction of
>>> the
>>>>>>                                                   quota_usages and
>>>>>>                                                   reservations
>>>>>>                                                   tables to cache
>>>>>>                                                   usage records is
>>> the
>>>>>>                                                   primary reason
>>> for
>>>>>>                                                   the race
>>>>>>                                                   problems in the
>>> Nova
>>>>>>                                                   (and
>>>>>>                                                   other) quota
>>> system
>>>>>>                                                   because every
>>> time
>>>>>>                                                   you introduce a
>>>>>>                                                   caching system
>>>>>>                                                   for
>>> highly-volatile
>>>>>>                                                   data (like usage
>>>>>>                                                   records) you
>>>>>>                                                   introduce
>>>>>>                                                   complexity into
>>> the
>>>>>>                                                   write path and
>>> the
>>>>>>                                                   need to track the
>>>>>>                                                   same thing
>>>>>>                                                   across multiple
>>>>>>                                                   writes to
>>> different
>>>>>>                                                   tables
>>> needlessly.
>>>>>>
>>>>>>                                           [amrith] I don't agree,
>>> I'll
>>>>>>                                           respond to this and the
>>> next
>>>>>>                                           comment group
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   together. See below.
>>>>>>
>>>>>>
>>>>>>                                                           Looking
>>> up
>>>>>>                                                           at this
>>>>>>                                                           whole
>>>>>>
>>> infrastructure from the perspective of the
>>>>>>                                                           database,
>>> I
>>>>>>                                                           think we
>>>>>>                                                           should
>>>>>>                                                           require
>>> that
>>>>>>                                                           the
>>> database
>>>>>>                                                           must not
>>> be
>>>>>>                                                           required
>>> to
>>>>>>                                                           operate
>>> in
>>>>>>                                                           any
>>>>>>                                                           isolation
>>>>>>                                                           mode
>>> higher
>>>>>>                                                           than
>>>>>>
>>> READ-COMMITTED; more about that later (i.e. requiring a database run
>>>>>>                                                           either
>>>>>>
>>> serializable
>>>>>>                                                           or
>>>>>>
>>> repeatable
>>>>>>                                                           read is a
>>>>>>                                                           show
>>>>>>                                                           stopper).
>>>>>>
>>>>>>
>>>>>>                                                   This is an
>>>>>>                                                   implementation
>>>>>>                                                   detail is not
>>>>>>                                                   relevant to the
>>>>>>                                                   discussion
>>>>>>                                                   about what the
>>>>>>                                                   interface of a
>>> quota
>>>>>>                                                   library would
>>> look
>>>>>>                                                   like.
>>>>>>
>>>>>>
>>>>>>                                           [amrith] I disagree, let
>>> me
>>>>>>                                           give you an example of
>>> why.
>>>>>>                                           Earlier, I wrote:
>>>>>>
>>>>>>                                                           Such a
>>>>>>
>>> hypothetical
>>>>>>                                                           service
>>> may
>>>>>>                                                           also
>>> consume
>>>>>>                                                           resources
>>>>>>                                                           from
>>> other
>>>>>>                                                           services
>>>>>>                                                           that it
>>>>>>                                                           wishes to
>>>>>>                                                           track,
>>> and
>>>>>>                                                           impose
>>>>>>                                                           limits
>>> on.
>>>>>>
>>>>>>                                           And you responded:
>>>>>>
>>>>>>
>>>>>>                                                   Yes, absolutely
>>>>>>                                                   agreed.
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                           So let's take this
>>>>>>                                           hypothetical service that
>>> in
>>>>>>                                           response to a user
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   request, will provision a Cinder
>>>>>>                                   volume and a Nova instance. Let's
>>>>>>                                   assume
>>>>>>                                   that the service also imposes
>>> limits
>>>>>>                                   on the number of cinder volumes
>>> and
>>>>>>                                   nova instances the user may
>>>>>>                                   provision; independent of limits
>>>>>>                                   that Nova and
>>>>>>                                   Cinder may themselves maintain.
>>>>>>
>>>>>>                                           One way that the
>>>>>>                                           hypothetical service can
>>>>>>                                           function is this:
>>>>>>
>>>>>>                                           (a) check Cinder quota,
>>> if
>>>>>>                                           successful, create cinder
>>>>>>                                           volume
>>>>>>                                           (b) check Nova quota, if
>>>>>>                                           successful, create nova
>>>>>>                                           instance with cinder
>>>>>>                                           volume attachment
>>>>>>
>>>>>>                                           Now, this is sub-optimal
>>> as
>>>>>>                                           there are going to be
>>> some
>>>>>>                                           number of cases
>>>>>>
>>>>>>
>>>>>>                                   where the nova quota check fails.
>>>>>>                                   Now you have needlessly created
>>> and
>>>>>>                                   will
>>>>>>                                   have to release a cinder volume.
>>> It
>>>>>>                                   also takes longer to fail.
>>>>>>
>>>>>>                                           Another way to do this is
>>>>>>                                           this:
>>>>>>
>>>>>>                                           (1) check Cinder quota,
>>> if
>>>>>>                                           successful, check Nova
>>>>>>                                           quota, if successful
>>>>>>                                           proceed to (2) else error
>>>>>>                                           out
>>>>>>                                           (2) create cinder volume
>>>>>>                                           (3) create nova instance
>>>>>>                                           with cinder attachment.
>>>>>>
>>>>>>                                           I'm trying to get to this
>>>>>>                                           latter form of doing
>>> things.
>>>>>>                                           Easy, you might say ...
>>>>>>                                           theoretically this should
>>>>>>                                           simply be:
>>>>>>
>>>>>>                                           BEGIN;
>>>>>>                                           -- Get data to do the
>>> Cinder
>>>>>>                                           check
>>>>>>
>>>>>>                                           SELECT ......
>>>>>>
>>>>>>                                           -- Do the cinder check
>>>>>>
>>>>>>                                           INSERT INTO ....
>>>>>>
>>>>>>                                           -- Get data to do the
>>> Nova
>>>>>>                                           check
>>>>>>
>>>>>>                                           SELECT ....
>>>>>>
>>>>>>                                           -- Do the Nova check
>>>>>>
>>>>>>                                           INSERT INTO ...
>>>>>>
>>>>>>                                           COMMIT
>>>>>>
>>>>>>                                           You can only make this
>>> work
>>>>>>                                           if you ran at isolation
>>>>>>                                           level serializable.
>>>>>>
>>>>>>
>>>>>>                                   Why?
>>>>>>
>>>>>>
>>>>>>                                           To make this run at
>>>>>>                                           isolation level
>>>>>>                                           REPEATABLE-READ, you must
>>>>>>                                           enforce
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   constraints at the database level
>>>>>>                                   that will fail the commit. But
>>> wait,
>>>>>>                                   you
>>>>>>                                   can't do that because the data
>>> about
>>>>>>                                   the global limits may not be in
>>> the
>>>>>>                                   same database as the usage
>>> records.
>>>>>>                                   Later you talk about caching and
>>>>>>                                   stuff; all that doesn't help a
>>>>>>                                   database constraint.
>>>>>>
>>>>>>                                           For this reason, I think
>>>>>>                                           there is going to have to
>>> be
>>>>>>                                           some cognizance to
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   the database isolation level in
>>> the
>>>>>>                                   design of the library, and I
>>> think
>>>>>>                                   it
>>>>>>                                   will also impact the API that can
>>> be
>>>>>>                                   constructed.
>>>>>>
>>>>>>                                                           In
>>> general
>>> therefore, I
>>>>>>                                                           believe
>>> that
>>>>>>                                                           the
>>>>>>
>>> hypothetical
>>>>>>                                                           service
>>>>>>
>>> processing
>>>>>>                                                           requests
>>> for
>>>>>>                                                           widgets
>>>>>>                                                           would
>>> have
>>>>>>                                                           to handle
>>>>>>                                                           three
>>> kinds
>>>>>>                                                           of
>>>>>>
>>> operations,
>>> provision,
>>>>>>                                                           modify,
>>> and
>>>>>>                                                           destroy.
>>> The
>>>>>>                                                           names
>>> are, I
>>>>>>                                                           believe,
>>>>>>
>>> self-explanatory.
>>>>>>
>>>>>>                                                   Generally,
>>>>>>                                                   modification of a
>>>>>>                                                   resource doesn't
>>>>>>                                                   come into play.
>>> The
>>>>>>                                                   primary exception
>>> to
>>>>>>                                                   this is for
>>>>>>                                                   transferring of
>>>>>>                                                   ownership of some
>>>>>>
>>>>>>
>>>>>>                                   resource.
>>>>>>
>>>>>>
>>>>>>                                           [amrith] Trove RESIZE is
>>> a
>>>>>>                                           huge benefit for users
>>> and
>>>>>>                                           while it may be a
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   pain as you say, this is still a
>>>>>>                                   very real benefit. Trove allows
>>> you
>>>>>>                                   to
>>>>>>                                   resize both your storage (resize
>>> the
>>>>>>                                   cinder volume) and resize your
>>>>>>                                   instance (change the flavor).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                                           Without
>>> loss
>>>>>>                                                           of
>>>>>>
>>> generality,
>>>>>>                                                           one can
>>> say
>>>>>>                                                           that all
>>>>>>                                                           three of
>>>>>>                                                           them must
>>>>>>                                                           validate
>>>>>>                                                           that the
>>>>>>                                                           operation
>>>>>>                                                           does not
>>>>>>                                                           violate
>>> some
>>>>>>                                                           limit (no
>>>>>>                                                           more
>>>>>>                                                           than X
>>>>>>                                                           widgets,
>>> no
>>>>>>                                                           fewer
>>> than X
>>>>>>                                                           widgets,
>>>>>>                                                           rates,
>>> and
>>>>>>                                                           so on).
>>>>>>
>>>>>>
>>>>>>                                                   No, only the
>>>>>>                                                   creation (and
>>> very
>>>>>>                                                   rarely the
>>>>>>                                                   modification)
>>> needs
>>>>>>                                                   any
>>>>>>                                                   validation that a
>>>>>>                                                   limit could been
>>>>>>                                                   violated.
>>> Destroying
>>>>>>                                                   a resource
>>>>>>                                                   never needs to be
>>>>>>                                                   checked for limit
>>>>>>                                                   violations.
>>>>>>
>>>>>>
>>>>>>                                           [amrith] Well, if you are
>>>>>>                                           going to create a volume
>>> of
>>>>>>                                           10GB and your
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   limit is 100GB, resizing it to
>>> 200GB
>>>>>>                                   should fail, I think.
>>>>>>
>>>>>>
>>>>>>                                                           Assuming
>>>>>>                                                           that the
>>>>>>                                                           service
>>>>>>
>>> provisions
>>>>>>                                                           resources
>>>>>>                                                           from
>>> other
>>>>>>                                                           services,
>>>>>>                                                           it is
>>> also
>>> conceivable
>>>>>>                                                           that
>>> limits
>>>>>>                                                           be
>>> imposed
>>>>>>                                                           on the
>>>>>>                                                           quantum
>>> of
>>>>>>                                                           those
>>>>>>                                                           services
>>>>>>                                                           consumed.
>>> In
>>>>>>                                                           practice,
>>> I
>>>>>>                                                           can
>>> imagine
>>>>>>                                                           a service
>>>>>>                                                           like
>>>>>>                                                           Trove
>>> using
>>>>>>                                                           the
>>>>>>                                                           Delimiter
>>>>>>                                                           project
>>> to
>>>>>>                                                           perform
>>> all
>>>>>>                                                           of these
>>>>>>                                                           kinds of
>>>>>>                                                           limit
>>>>>>                                                           checks;
>>> I'm
>>>>>>                                                           not
>>>>>>
>>> suggesting
>>>>>>                                                           that it
>>> does
>>>>>>                                                           this
>>> today,
>>>>>>                                                           nor that
>>>>>>                                                           there is
>>> an
>>>>>>                                                           immediate
>>>>>>                                                           plan to
>>>>>>                                                           implement
>>>>>>                                                           all of
>>> them,
>>>>>>                                                           just that
>>>>>>                                                           these
>>>>>>                                                           all seem
>>>>>>                                                           like good
>>>>>>                                                           uses a
>>> Quota
>>> Management
>>> capability.
>>>>>>                                                           - User
>>> may
>>>>>>                                                           not have
>>>>>>                                                           more than
>>> 25
>>>>>>                                                           database
>>>>>>                                                           instances
>>> at
>>>>>>                                                           a
>>>>>>
>>>>>>
>>>>>>                                   time
>>>>>>
>>>>>>
>>>>>>
>>> -
>>>>>>                                                           User may
>>> not
>>>>>>                                                           have more
>>>>>>                                                           than 4
>>>>>>                                                           clusters
>>> at
>>>>>>                                                           a time
>>>>>>                                                           - User
>>> may
>>>>>>                                                           not
>>> consume
>>>>>>                                                           more than
>>>>>>                                                           3TB of
>>> SSD
>>>>>>                                                           storage
>>> at a
>>>>>>                                                           time
>>>>>>
>>>>>>
>>>>>>                                                   Only if SSD
>>> storage
>>>>>>                                                   is a distinct
>>>>>>                                                   resource class
>>> from
>>>>>>                                                   DISK_GB. Right
>>>>>>                                                   now, Nova makes
>>> no
>>>>>>                                                   differentiation
>>>>>>                                                   w.r.t. SSD or HDD
>>> or
>>>>>>                                                   shared vs.
>>>>>>                                                   local block
>>> storage.
>>>>>>
>>>>>>                                           [amrith] It matters not
>>> to
>>>>>>                                           Trove whether Nova does
>>> nor
>>>>>>                                           not. Cinder
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   supports volume-types and users
>>> DO
>>>>>>                                   want to limit based on
>>> volume-type
>>>>>>                                   (for
>>>>>>                                   example).
>>>>>>
>>>>>>
>>> -
>>>>>>                                                           User may
>>> not
>>>>>>                                                           launch
>>> more
>>>>>>                                                           than 10
>>> huge
>>>>>>                                                           instances
>>> at
>>>>>>                                                           a
>>>>>>                                                           time
>>>>>>
>>>>>>
>>>>>>                                                   What is the point
>>> of
>>>>>>                                                   such a limit?
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                           [amrith] Metering usage,
>>>>>>                                           placing limitations on
>>> the
>>>>>>                                           quantum of resources
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   that a user may provision. Same
>>> as
>>>>>>                                   with Nova. A flavor is merely a
>>>>>>                                   simple
>>>>>>                                   way to tie together a bag of
>>>>>>                                   resources. It is a way to
>>> restrict
>>>>>>                                   access,
>>>>>>                                   for example, to specific
>>> resources
>>>>>>                                   that are available in the cloud.
>>>>>>                                   HUGE
>>>>>>                                   is just an example I gave, pick
>>> any
>>>>>>                                   flavor you want, and here's how a
>>>>>>                                   service like Trove uses it.
>>>>>>
>>>>>>                                           Users can ask to launch
>>> an
>>>>>>                                           instance of a specific
>>>>>>                                           database+version;
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   MySQL 5.6-48 for example. Now, an
>>>>>>                                   operator can restrict the
>>> instance
>>>>>>                                   flavors, or volume types that can
>>> be
>>>>>>                                   associated with the specific
>>>>>>                                   datastore. And the flavor could
>>> be
>>>>>>                                   used to map to, for example
>>> whether
>>>>>>                                   the
>>>>>>                                   instance is running on bare metal
>>> or
>>>>>>                                   in a VM and if so with what kind
>>> of
>>>>>>                                   hardware. That's a useful
>>> construct
>>>>>>                                   for a service like Trove.
>>>>>>
>>>>>>
>>> -
>>>>>>                                                           User may
>>> not
>>>>>>                                                           launch
>>> more
>>>>>>                                                           than 3
>>>>>>                                                           clusters
>>> an
>>>>>>                                                           hour
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                                   -1. This is rate
>>>>>>                                                   limiting and
>>> should
>>>>>>                                                   be handled by
>>>>>>                                                   rate-limiting
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   services.
>>>>>>
>>>>>>
>>>>>>
>>> -
>>>>>>                                                           No more
>>> than
>>>>>>                                                           500
>>> copies
>>>>>>                                                           of Oracle
>>>>>>                                                           may be
>>> run
>>>>>>                                                           at a time
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                                   Is "Oracle" a
>>>>>>                                                   resource class?
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                           [amrith] As I view it,
>>> every
>>>>>>                                           project should be free to
>>>>>>                                           define its own
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   set of resource classes and meter
>>>>>>                                   them as it feels fit. So, while
>>>>>>                                   Oracle
>>>>>>                                   licenses may not, conceivably a
>>> lot
>>>>>>                                   of things that Nova, Cinder, and
>>> the
>>>>>>                                   other core projects don't care
>>>>>>                                   about, are in fact relevant for a
>>>>>>                                   consumer
>>>>>>                                   of this library.
>>>>>>
>>>>>>                                                           While
>>> Nova
>>>>>>                                                           would be
>>> the
>>>>>>                                                           service
>>> that
>>>>>>                                                           limits
>>> the
>>>>>>                                                           number of
>>>>>>                                                           instances
>>>>>>                                                           a user
>>> can
>>>>>>                                                           have at a
>>>>>>                                                           time, the
>>>>>>                                                           ability
>>> for
>>>>>>                                                           a service
>>> to
>>>>>>                                                           limit
>>> this
>>>>>>                                                           further
>>>>>>                                                           should
>>> not
>>>>>>                                                           be
>>>>>>
>>> underestimated.
>>>>>>                                                           In turn,
>>>>>>                                                           should
>>> Nova
>>>>>>                                                           and
>>> Cinder
>>>>>>                                                           also use
>>> the
>>>>>>                                                           same
>>> Quota
>>> Management
>>>>>>                                                           Library,
>>>>>>                                                           they may
>>>>>>                                                           each
>>> impose
>>> limitations
>>>>>>                                                           like:
>>>>>>
>>>>>>                                                           - User
>>> may
>>>>>>                                                           not
>>> launch
>>>>>>                                                           more than
>>> 20
>>>>>>                                                           huge
>>>>>>                                                           instances
>>> at
>>>>>>                                                           a
>>>>>>                                                           time
>>>>>>
>>>>>>
>>>>>>                                                   Not a useful
>>>>>>                                                   limitation IMHO.
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                           [amrith] I beg to differ.
>>>>>>                                           Again a huge instance is
>>>>>>                                           just an example of
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   some flavor; and the idea is to
>>>>>>                                   allow a project to place its own
>>>>>>                                   metrics
>>>>>>                                   and meter based on those.
>>>>>>
>>>>>>
>>> -
>>>>>>                                                           User may
>>> not
>>>>>>                                                           launch
>>> more
>>>>>>                                                           than 3
>>>>>>                                                           instances
>>> in
>>>>>>                                                           a minute
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                                   -1. This is rate
>>>>>>                                                   limiting.
>>>>>>
>>>>>>
>>>>>>
>>> -
>>>>>>                                                           User may
>>> not
>>>>>>                                                           consume
>>> more
>>>>>>                                                           than 15TB
>>> of
>>>>>>                                                           SSD at a
>>>>>>                                                           time
>>>>>>                                                           - User
>>> may
>>>>>>                                                           not have
>>>>>>                                                           more than
>>> 30
>>>>>>                                                           volumes
>>> at a
>>>>>>                                                           time
>>>>>>
>>>>>>                                                           Again,
>>> I'm
>>>>>>                                                           not
>>> implying
>>>>>>                                                           that
>>> either
>>>>>>                                                           Nova or
>>>>>>                                                           Cinder
>>>>>>                                                           should
>>>>>>                                                           provide
>>>>>>                                                           these
>>>>>>
>>> capabilities.
>>>>>>                                                           With this
>>> in
>>>>>>                                                           mind, I
>>>>>>                                                           believe
>>> that
>>>>>>                                                           the
>>> minimal
>>>>>>                                                           set of
>>>>>>
>>> operations
>>>>>>                                                           that
>>>>>>                                                           Delimiter
>>>>>>                                                           should
>>>>>>                                                           provide
>>> are:
>>>>>>                                                           -
>>>>>>
>>> define_resource(name, max, min, user_max, user_min, ...)
>>>>>>
>>>>>>                                                   What would the
>>> above
>>>>>>                                                   do? What service
>>>>>>                                                   would it be
>>> speaking
>>>>>>                                                   to?
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                           [amrith] I assume that
>>> this
>>>>>>                                           would speak with some
>>>>>>                                           backend (either
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   keystone or the project itself)
>>> and
>>>>>>                                   record these designated limits.
>>> This
>>>>>>                                   is the way to register a project
>>>>>>                                   specific metric like "Oracle
>>>>>>                                   licenses".
>>>>>>
>>>>>>
>>> -
>>> update_resource_limits(name, user, user_max, user_min,
>>>>>>                                                           ...)
>>>>>>
>>>>>>
>>>>>>                                                   This doesn't
>>> belong
>>>>>>                                                   in a quota
>>> library.
>>>>>>                                                   It belongs as a
>>> REST
>>>>>>                                                   API in
>>>>>>                                                   Keystone.
>>>>>>
>>>>>>
>>>>>>                                           [amrith] Fine, same place
>>>>>>                                           where the previous thing
>>>>>>                                           stores the global
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   defaults is the target of this
>>> call.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> -
>>> reserve_resource(name, user, size, parent_resource, ...)
>>>>>>
>>>>>>
>>>>>>                                                   This doesn't
>>> belong
>>>>>>                                                   in a quota
>>> library
>>>>>>                                                   at all. I think
>>>>>>                                                   reservations
>>>>>>                                                   are not germane
>>> to
>>>>>>                                                   resource
>>> consumption
>>>>>>                                                   and should be
>>>>>>                                                   handled by an
>>>>>>                                                   external service
>>> at
>>>>>>                                                   the orchestration
>>>>>>                                                   layer.
>>>>>>
>>>>>>
>>>>>>                                           [amrith] Again not true,
>>> as
>>>>>>                                           illustrated above this
>>>>>>                                           library is the thing
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   that projects could use to
>>> determine
>>>>>>                                   whether or not to honor a
>>> request.
>>>>>>                                   This reserve/provision process
>>> is, I
>>>>>>                                   believe required because of the
>>>>>>                                   vagaries of how we want to
>>> implement
>>>>>>                                   this in the database.
>>>>>>
>>>>>>
>>> -
>>> provision_resource(resource, id)
>>>>>>
>>>>>>
>>>>>>                                                   A quota library
>>>>>>                                                   should not be
>>>>>>                                                   provisioning
>>>>>>                                                   anything. A quota
>>>>>>                                                   library
>>>>>>                                                   should simply
>>>>>>                                                   provide a
>>> consistent
>>>>>>                                                   interface for
>>>>>>                                                   *checking* that a
>>>>>>                                                   structured
>>> request
>>>>>>                                                   for some set of
>>>>>>                                                   resources *can*
>>> be
>>>>>>                                                   provided by the
>>>>>>                                                   service.
>>>>>>
>>>>>>
>>>>>>                                           [amrith] This does not
>>>>>>                                           actually call Nova or
>>>>>>                                           anything; merely that
>>> AFTER
>>>>>>
>>>>>>
>>>>>>                                   the hypothetical service has
>>> called
>>>>>>                                   NOVA, this converts the
>>> reservation
>>>>>>                                   (which can expire) into an actual
>>>>>>                                   allocation.
>>>>>>
>>>>>>
>>> -
>>> update_resource(id or resource, newsize)
>>>>>>
>>>>>>
>>>>>>                                                   Resizing
>>> resources
>>>>>>                                                   is a bad idea,
>>> IMHO.
>>>>>>                                                   Resources are
>>> easier
>>>>>>                                                   to deal
>>>>>>                                                   with when they
>>> are
>>>>>>                                                   considered of
>>>>>>                                                   immutable size
>>> and
>>>>>>                                                   simple (i.e. not
>>>>>>                                                   complex or
>>> nested).
>>>>>>                                                   I think the
>>> problem
>>>>>>                                                   here is in the
>>>>>>                                                   definition of
>>>>>>                                                   resource classes
>>>>>>                                                   improperly.
>>>>>>
>>>>>>
>>>>>>                                           [amrith] Let's leave the
>>>>>>                                           quota library aside. This
>>>>>>                                           assertion strikes at
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   the very heart of things like
>>> Nova
>>>>>>                                   resize, or for that matter Cinder
>>>>>>                                   volume resize. Are those all bad
>>>>>>                                   ideas? I made a 500GB Cinder
>>> volume
>>>>>>                                   and
>>>>>>                                   it is getting close to full. I'd
>>>>>>                                   like to resize it to 2TB. Are you
>>>>>>                                   saying
>>>>>>                                   that's not a valid use case?
>>>>>>
>>>>>>                                                   For example, a
>>>>>>                                                   "cluster" is not
>>> a
>>>>>>                                                   resource. It is a
>>>>>>                                                   collection of
>>>>>>                                                   resources of type
>>>>>>                                                   node. "Resizing"
>>> a
>>>>>>                                                   cluster is a
>>>>>>                                                   misnomer, because
>>>>>>                                                   you aren't
>>> resizing
>>>>>>                                                   a resource at
>>> all.
>>>>>>                                                   Instead, you are
>>>>>>                                                   creating or
>>>>>>                                                   destroying
>>> resources
>>>>>>                                                   inside the
>>> cluster
>>>>>>                                                   (i.e. joining or
>>>>>>                                                   leaving
>>>>>>
>>>>>>
>>>>>>                                   cluster nodes).
>>>>>>
>>>>>>
>>>>>>                                                   BTW, this is also
>>>>>>                                                   why the "resize
>>>>>>                                                   instance" API in
>>>>>>                                                   Nova is such a
>>>>>>                                                   giant pain in the
>>>>>>                                                   ass. It's
>>> attempting
>>>>>>                                                   to "modify" the
>>>>>>                                                   instance
>>>>>>
>>>>>>
>>>>>>                                   "resource"
>>>>>>
>>>>>>
>>>>>>                                                   when the instance
>>>>>>                                                   isn't really the
>>>>>>                                                   resource at all.
>>> The
>>>>>>                                                   VCPU, RAM_MB,
>>>>>>                                                   DISK_GB, and PCI
>>>>>>                                                   devices are the
>>>>>>                                                   actual resources.
>>>>>>                                                   The instance is a
>>>>>>                                                   convenient way to
>>>>>>                                                   tie those
>>> resources
>>>>>>                                                   together, and
>>> doing
>>>>>>                                                   a "resize"
>>>>>>                                                   of the instance
>>>>>>                                                   behind the scenes
>>>>>>                                                   actually performs
>>> a
>>>>>>                                                   *move*
>>>>>>                                                   operation, which
>>>>>>                                                   isn't a *change*
>>> of
>>>>>>                                                   the original
>>>>>>                                                   resources.
>>> Rather,
>>>>>>                                                   it is a creation
>>> of
>>>>>>                                                   a new set of
>>>>>>                                                   resources (of the
>>>>>>                                                   new amounts) and
>>> a
>>>>>>                                                   deletion of the
>>> old
>>>>>>                                                   set of resources.
>>>>>>
>>>>>>
>>>>>>                                           [amrith] that's fine, if
>>> all
>>>>>>                                           we want is to handle the
>>>>>>                                           resize operation
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   as a new instance followed by a
>>>>>>                                   deletion, that's great. But that
>>>>>>                                   semantic
>>>>>>                                   isn't necessarily the case for
>>>>>>                                   something like (say) cinder.
>>>>>>
>>>>>>                                                   The "resize" API
>>>>>>                                                   call adds some
>>> nasty
>>>>>>                                                   confirmation and
>>>>>>                                                   cancel
>>>>>>                                                   semantics to the
>>>>>>                                                   calling interface
>>>>>>                                                   that hint that
>>> the
>>>>>>                                                   underlying
>>>>>>                                                   implementation of
>>>>>>                                                   the "resize"
>>>>>>                                                   operation is in
>>>>>>                                                   actuality not a
>>>>>>                                                   resize
>>>>>>                                                   at all, but
>>> rather a
>>> create-new-and-delete-old-resources operation.
>>>>>>
>>>>>>                                           [amrith] And that isn't
>>>>>>                                           germane to a quota
>>> library,
>>>>>>                                           I don't think. What
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   is, is this. Do we want to treat
>>> the
>>>>>>                                   transient state when there are
>>> (for
>>>>>>                                   example of Nova) two instances,
>>> one
>>>>>>                                   of the new flavor and one of the
>>> old
>>>>>>                                   flavor, or not. But, from the
>>>>>>                                   perspective of a quota library, a
>>>>>>                                   resize
>>>>>>                                   operation is merely a reset of
>>> the
>>>>>>                                   quota by the delta in the
>>> resource
>>>>>>                                   consumed.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> -
>>> release_resource(id or resource)
>>>>>>                                                           -
>>>>>>
>>> expire_reservations()
>>>>>>
>>>>>>                                                   I see no need to
>>>>>>                                                   have reservations
>>> in
>>>>>>                                                   the quota library
>>> at
>>>>>>                                                   all, as
>>>>>>                                                   mentioned above.
>>>>>>
>>>>>>
>>>>>>                                           [amrith] Then I think the
>>>>>>                                           quota library must
>>> require
>>>>>>                                           that either (a) the
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   underlying database runs
>>>>>>                                   serializable or (b) database
>>>>>>                                   constraints can be
>>>>>>                                   used to enforce that at commit
>>> the
>>>>>>                                   global limits are adhered to.
>>>>>>
>>>>>>                                                   As for your
>>> proposed
>>>>>>                                                   interface and
>>>>>>                                                   calling structure
>>>>>>                                                   below, I think a
>>>>>>                                                   much simpler
>>>>>>                                                   proposal would
>>> work
>>>>>>                                                   better. I'll work
>>> on
>>>>>>                                                   a cross-project
>>>>>>                                                   spec that
>>> describes
>>>>>>                                                   this simpler
>>>>>>                                                   proposal, but the
>>>>>>                                                   basics would be:
>>>>>>
>>>>>>                                                   1) Have Keystone
>>>>>>                                                   store quota
>>>>>>                                                   information for
>>>>>>                                                   defaults (per
>>>>>>                                                   service
>>>>>>                                                   endpoint), for
>>>>>>                                                   tenants and for
>>>>>>                                                   users.
>>>>>>
>>>>>>                                                   Keystone would
>>> have
>>>>>>                                                   the set of
>>> canonical
>>>>>>                                                   resource class
>>>>>>                                                   names, and
>>>>>>                                                   each project,
>>> upon
>>>>>>                                                   handling a new
>>>>>>                                                   resource class,
>>>>>>                                                   would be
>>>>>>                                                   responsible for a
>>>>>>                                                   change submitted
>>> to
>>>>>>                                                   Keystone to add
>>> the
>>>>>>                                                   new resource
>>>>>>
>>>>>>
>>>>>>                                   class code.
>>>>>>
>>>>>>
>>>>>>                                                   Straw man REST
>>> API:
>>>>>>
>>> GET /quotas/resource-classes
>>>>>>                                                   200 OK
>>>>>>                                                   {
>>>>>>
>>> "resource_classes":
>>>>>>                                                   {
>>>>>>                                                   "compute.vcpu": {
>>>>>>                                                   "service":
>>>>>>                                                   "compute",
>>>>>>                                                   "code":
>>>>>>                                                   "compute.vcpu",
>>>>>>                                                   "description": "A
>>>>>>                                                   virtual CPU unit"
>>>>>>                                                   },
>>>>>>                                                   "compute.ram_mb":
>>> {
>>>>>>                                                   "service":
>>>>>>                                                   "compute",
>>>>>>                                                   "code":
>>>>>>                                                   "compute.ram_mb",
>>>>>>                                                   "description":
>>>>>>                                                   "Memory in
>>>>>>                                                   megabytes"
>>>>>>                                                   },
>>>>>>                                                   ...
>>>>>>                                                   "volume.disk_gb":
>>> {
>>>>>>                                                   "service":
>>> "volume",
>>>>>>                                                   "code":
>>>>>>                                                   "volume.disk_gb",
>>>>>>                                                   "description":
>>>>>>                                                   "Amount of disk
>>>>>>                                                   space in
>>> gigabytes"
>>>>>>                                                   },
>>>>>>                                                   ...
>>>>>>                                                   "database.count":
>>> {
>>>>>>                                                   "service":
>>>>>>                                                   "database",
>>>>>>                                                   "code":
>>>>>>                                                   "database.count",
>>>>>>                                                   "description":
>>>>>>                                                   "Number of
>>> database
>>>>>>                                                   instances"
>>>>>>                                                   }
>>>>>>                                                   }
>>>>>>                                                   }
>>>>>>
>>>>>>
>>>>>>                                           [amrith] Well, a user is
>>>>>>                                           allowed to have a certain
>>>>>>                                           compute quota (which
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   is shared by Nova and Trove) but
>>>>>>                                   also a Trove quota. How would
>>> your
>>>>>>                                   representation represent that?
>>>>>>
>>>>>>                                                   # Get the default
>>>>>>                                                   limits for new
>>>>>>                                                   users...
>>>>>>
>>> GET /quotas/defaults
>>>>>>                                                   200 OK
>>>>>>                                                   {
>>>>>>                                                   "quotas": {
>>>>>>                                                   "compute.vcpu":
>>> 100,
>>>>>>                                                   "compute.ram_mb":
>>>>>>                                                   32768,
>>>>>>                                                   "volume.disk_gb":
>>>>>>                                                   1000,
>>>>>>                                                   "database.count":
>>> 25
>>>>>>                                                   }
>>>>>>                                                   }
>>>>>>
>>>>>>                                                   # Get a specific
>>>>>>                                                   user's limits...
>>>>>>
>>> GET /quotas/users/{UUID}
>>>>>>                                                   200 OK
>>>>>>                                                   {
>>>>>>                                                   "quotas": {
>>>>>>                                                   "compute.vcpu":
>>> 100,
>>>>>>                                                   "compute.ram_mb":
>>>>>>                                                   32768,
>>>>>>                                                   "volume.disk_gb":
>>>>>>                                                   1000,
>>>>>>                                                   "database.count":
>>> 25
>>>>>>                                                   }
>>>>>>                                                   }
>>>>>>
>>>>>>                                                   # Get a tenant's
>>>>>>                                                   limits...
>>>>>>
>>> GET /quotas/tenants/{UUID}
>>>>>>                                                   200 OK
>>>>>>                                                   {
>>>>>>                                                   "quotas": {
>>>>>>                                                   "compute.vcpu":
>>>>>>                                                   1000,
>>>>>>                                                   "compute.ram_mb":
>>>>>>                                                   327680,
>>>>>>                                                   "volume.disk_gb":
>>>>>>                                                   10000,
>>>>>>                                                   "database.count":
>>>>>>                                                   250
>>>>>>                                                   }
>>>>>>                                                   }
>>>>>>
>>>>>>                                                   2) Have Delimiter
>>>>>>                                                   communicate with
>>> the
>>>>>>                                                   above proposed
>>> new
>>>>>>                                                   Keystone
>>>>>>                                                   REST API and
>>> package
>>>>>>                                                   up data into an
>>>>>>
>>> oslo.versioned_objects interface.
>>>>>>                                                   Clearly all of
>>> the
>>>>>>                                                   above can be
>>> heavily
>>>>>>                                                   cached both on
>>> the
>>>>>>                                                   server and
>>>>>>                                                   client side since
>>>>>>                                                   they rarely
>>> change
>>>>>>                                                   but are read
>>> often.
>>>>>>
>>>>>>                                           [amrith] Caching on the
>>>>>>                                           client won't save you
>>> from
>>>>>>                                           oversubscription if
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   you don't run serializable.
>>>>>>
>>>>>>
>>>>>>                                                   The Delimiter
>>>>>>                                                   library could be
>>>>>>                                                   used to provide a
>>>>>>                                                   calling interface
>>>>>>                                                   for service
>>> projects
>>>>>>                                                   to get a user's
>>>>>>                                                   limits for a set
>>> of
>>>>>>                                                   resource
>>>>>>
>>>>>>
>>>>>>                                   classes:
>>>>>>
>>>>>>
>>>>>>                                                   (please excuse
>>>>>>                                                   wrongness, typos,
>>>>>>                                                   and other stuff
>>>>>>                                                   below, it's just
>>> a
>>>>>>                                                   straw- man not
>>>>>>                                                   production
>>> working
>>>>>>                                                   code...)
>>>>>>
>>>>>>                                                   # file:
>>>>>>
>>> delimiter/objects/limits.py
>>>>>>                                                   import
>>>>>>
>>> oslo.versioned_objects.base as ovo import
>>> oslo.versioned_objects.fields as ovo_fields
>>>>>>
>>>>>>                                                   class
>>>>>>
>>> ResourceLimit(ovo.VersionedObjectBase):
>>>>>>                                                   # 1.0: Initial
>>>>>>                                                   version
>>>>>>                                                   VERSION = '1.0'
>>>>>>
>>>>>>                                                   fields = {
>>>>>>                                                   'resource_class':
>>>>>>
>>> ovo_fields.StringField(),
>>>>>>                                                   'amount':
>>>>>>
>>> ovo_fields.IntegerField(),
>>>>>>                                                   }
>>>>>>
>>>>>>
>>>>>>                                                   class
>>>>>>
>>> ResourceLimitList(ovo.VersionedObjectBase):
>>>>>>                                                   # 1.0: Initial
>>>>>>                                                   version
>>>>>>                                                   VERSION = '1.0'
>>>>>>
>>>>>>                                                   fields = {
>>>>>>                                                   'resources':
>>>>>>
>>> ListOfObjectsField(ResourceLimit),
>>>>>>                                                   }
>>>>>>
>>>>>>
>>> @cache_this_heavily
>>> @remotable_classmethod
>>>>>>                                                   def
>>>>>>
>>> get_all_by_user(cls,
>>>>>>                                                   user_uuid):
>>>>>>                                                   """Returns a
>>> Limits
>>>>>>                                                   object that tells
>>>>>>                                                   the caller what a
>>>>>>                                                   user's
>>>>>>                                                   absolute limits
>>> for
>>>>>>                                                   the set of
>>> resource
>>>>>>                                                   classes in the
>>>>>>                                                   system.
>>>>>>                                                   """
>>>>>>                                                   # Grab a keystone
>>>>>>                                                   client session
>>>>>>                                                   object and
>>> connect
>>>>>>                                                   to Keystone
>>>>>>                                                   ks =
>>>>>>
>>> ksclient.Session(...)
>>>>>>                                                   raw_limits =
>>>>>>
>>> ksclient.get_limits_by_user()
>>>>>>                                                   return
>>>>>>
>>> cls(resources=[ResourceLimit(**d) for d in raw_limits])
>>>>>>                                                   3) Each service
>>>>>>                                                   project would be
>>>>>>                                                   responsible for
>>>>>>                                                   handling the
>>>>>>                                                   consumption of a
>>> set
>>>>>>                                                   of requested
>>>>>>                                                   resource amounts
>>> in
>>>>>>                                                   an atomic and
>>>>>>
>>>>>>
>>>>>>                                   consistent way.
>>>>>>
>>>>>>
>>>>>>                                           [amrith] This is where
>>> the
>>>>>>                                           rubber meets the road.
>>> What
>>>>>>                                           is that atomic
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   and consistent way? And what
>>>>>>                                   computing infrastructure do you
>>> need
>>>>>>                                   to
>>>>>>                                   deliver this?
>>>>>>
>>>>>>                                                   The Delimiter
>>>>>>                                                   library would
>>> return
>>>>>>                                                   the limits that
>>> the
>>>>>>                                                   service would
>>>>>>                                                   pre- check before
>>>>>>                                                   claiming the
>>>>>>                                                   resources and
>>> either
>>>>>>                                                   post-check after
>>>>>>                                                   claim or utilize
>>> a
>>> compare-and-update
>>>>>>                                                   technique with a
>>>>>>
>>> generation/timestamp
>>>>>>                                                   during claiming
>>> to
>>>>>>                                                   prevent race
>>>>>>                                                   conditions.
>>>>>>
>>>>>>                                                   For instance, in
>>>>>>                                                   Nova with the new
>>>>>>                                                   resource
>>> providers
>>>>>>                                                   database schema
>>>>>>                                                   and doing claims
>>> in
>>>>>>                                                   the scheduler (a
>>>>>>                                                   proposed change),
>>> we
>>>>>>                                                   might do
>>>>>>                                                   something to the
>>>>>>                                                   effect of:
>>>>>>
>>>>>>                                                   from delimiter
>>>>>>                                                   import objects as
>>>>>>                                                   delim_obj from
>>>>>>                                                   delimier import
>>>>>>                                                   exceptions as
>>>>>>                                                   delim_exc from
>>> nova
>>>>>>                                                   import objects as
>>>>>>                                                   nova_obj
>>>>>>
>>>>>>                                                   request =
>>>>>>
>>> nova_obj.RequestSpec.get_by_uuid(request_uuid)
>>>>>>                                                   requested =
>>>>>>                                                   request.resources
>>>>>>                                                   limits =
>>>>>>
>>> delim_obj.ResourceLimitList.get_all_by_user(user_uuid)
>>>>>>                                                   allocations =
>>>>>>
>>> nova_obj.AllocationList.get_all_by_user(user_uuid)
>>>>>>                                                   # Pre-check for
>>>>>>                                                   violations
>>>>>>                                                   for
>>> resource_class,
>>>>>>                                                   requested_amount
>>> in
>>> requested.items():
>>>>>>                                                   limit_idx =
>>>>>>
>>> limits.resources.index(resource_class)
>>>>>>                                                   resource_limit =
>>>>>>
>>> limits.resources[limit_idx].amount
>>>>>>                                                   alloc_idx =
>>>>>>
>>> allocations.resources.index(resource_class)
>>>>>>                                                   resource_used =
>>>>>>
>>> allocations.resources[alloc_idx]
>>>>>>                                                   if (resource_used
>>> +
>>> requested_amount)>
>>>>>>                                                   resource_limit:
>>>>>>                                                   raise
>>>>>>
>>> delim_exc.QuotaExceeded
>>>>>>
>>>>>>                                           [amrith] Is the above
>>> code
>>>>>>                                           run with some global
>>> mutex
>>>>>>                                           to prevent that
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   two people don't believe that
>>> they
>>>>>>                                   are good on quota at the same
>>> time?
>>>>>>
>>>>>>                                                   # Do claims in
>>>>>>                                                   scheduler in an
>>>>>>                                                   atomic,
>>> consistent
>>>>>>                                                   fashion...
>>>>>>                                                   claims =
>>>>>>
>>> scheduler_client.claim_resources(request)
>>>>>>
>>>>>>                                           [amrith] Yes, each
>>> 'atomic'
>>>>>>                                           claim on a
>>> repeatable-read
>>>>>>                                           database could
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   result in oversubscription.
>>>>>>
>>>>>>
>>>>>>                                                   # Post-check for
>>>>>>                                                   violations
>>>>>>                                                   allocations =
>>>>>>
>>> nova_obj.AllocationList.get_all_by_user(user_uuid)
>>>>>>                                                   # allocations now
>>>>>>                                                   include the
>>> claimed
>>>>>>                                                   resources from
>>> the
>>>>>>                                                   scheduler
>>>>>>
>>>>>>                                                   for
>>> resource_class,
>>>>>>                                                   requested_amount
>>> in
>>> requested.items():
>>>>>>                                                   limit_idx =
>>>>>>
>>> limits.resources.index(resource_class)
>>>>>>                                                   resource_limit =
>>>>>>
>>> limits.resources[limit_idx].amount
>>>>>>                                                   alloc_idx =
>>>>>>
>>> allocations.resources.index(resource_class)
>>>>>>                                                   resource_used =
>>>>>>
>>> allocations.resources[alloc_idx]
>>>>>>                                                   if resource_used>
>>>>>>                                                   resource_limit:
>>>>>>                                                   # Delete the
>>>>>>                                                   allocation
>>> records
>>>>>>                                                   for the resources
>>>>>>                                                   just claimed
>>>>>>
>>> delete_resources(claims)
>>>>>>                                                   raise
>>>>>>
>>> delim_exc.QuotaExceeded
>>>>>>
>>>>>>                                           [amrith] Again, two
>>> people
>>>>>>                                           could drive through this
>>>>>>                                           code and both of
>>>>>>                                           them could fail :(
>>>>>>
>>>>>>                                                   4) The only other
>>>>>>                                                   thing that would
>>>>>>                                                   need to be done
>>> for
>>>>>>                                                   a first go of
>>>>>>                                                   the Delimiter
>>>>>>                                                   library is some
>>>>>>                                                   event listener
>>> that
>>>>>>                                                   can listen for
>>>>>>                                                   changes to the
>>> quota
>>>>>>                                                   limits for a
>>>>>>
>>> user/tenant/default
>>>>>>                                                   in Keystone.
>>>>>>                                                   We'd want the
>>>>>>                                                   services to be
>>> able
>>>>>>                                                   notify someone if
>>> a
>>>>>>                                                   reduction in
>>>>>>                                                   quota results in
>>> an
>>>>>>                                                   overquota
>>> situation.
>>>>>>                                                   Anyway, that's my
>>>>>>                                                   idea. Keep the
>>>>>>                                                   Delimiter library
>>>>>>                                                   small and focused
>>>>>>                                                   on describing the
>>>>>>                                                   limits only, not
>>> on
>>>>>>                                                   the resource
>>>>>>                                                   allocations. Have
>>>>>>                                                   the Delimiter
>>>>>>                                                   library present a
>>>>>>                                                   versioned object
>>>>>>                                                   interface so the
>>>>>>                                                   interaction
>>> between
>>>>>>                                                   the data exposed
>>> by
>>>>>>                                                   the Keystone REST
>>>>>>                                                   API for
>>>>>>                                                   quotas can evolve
>>>>>>                                                   naturally and
>>>>>>                                                   smoothly over
>>> time.
>>>>>>                                                   Best,
>>>>>>                                                   -jay
>>>>>>
>>>>>>                                                           Let me
>>>>>>
>>> illustrate
>>>>>>                                                           the way I
>>>>>>                                                           see these
>>>>>>                                                           things
>>>>>>                                                           fitting
>>>>>>                                                           together.
>>> A
>>> hypothetical
>>>>>>                                                           Trove
>>> system
>>>>>>                                                           may be
>>> setup
>>>>>>                                                           as
>>> follows:
>>>>>>                                                           - No more
>>>>>>                                                           than 2000
>>>>>>                                                           database
>>>>>>                                                           instances
>>> in
>>>>>>                                                           total,
>>> 300
>>>>>>                                                           clusters
>>>>>>
>>>>>>
>>>>>>                                                   in
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                                            total
>>>>>>                                                           - Users
>>> may
>>>>>>                                                           not
>>> launch
>>>>>>                                                           more than
>>> 25
>>>>>>                                                           database
>>>>>>
>>> instances,
>>>>>>                                                           or 4
>>>>>>                                                           clusters
>>>>>>                                                           - The
>>>>>>
>>> particular
>>>>>>                                                           user
>>>>>>                                                           'amrith'
>>> is
>>>>>>                                                           limited
>>> to 2
>>>>>>                                                           databases
>>>>>>                                                           and
>>>>>>
>>>>>>
>>>>>>                                   1
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                                            cluster
>>>>>>                                                           - No user
>>>>>>                                                           may
>>> consume
>>>>>>                                                           more than
>>>>>>                                                           20TB of
>>>>>>                                                           storage
>>> at a
>>>>>>                                                           time
>>>>>>                                                           - No user
>>>>>>                                                           may
>>> consume
>>>>>>                                                           more than
>>>>>>                                                           10GB of
>>>>>>                                                           memory at
>>> a
>>>>>>                                                           time
>>>>>>
>>>>>>                                                           At
>>> startup,
>>>>>>                                                           I believe
>>>>>>                                                           that the
>>>>>>                                                           system
>>> would
>>>>>>                                                           make the
>>>>>>                                                           following
>>>>>>                                                           sequence
>>> of
>>>>>>                                                           calls:
>>>>>>
>>>>>>                                                           -
>>>>>>
>>> define_resource(databaseInstance, 2000, 0, 25, 0, ...)
>>>>>>                                                           -
>>>>>>
>>> update_resource_limits(databaseInstance, amrith, 2, 0,
>>>>>>
>>>>>>                                   ...)
>>>>>>
>>>>>>
>>>>>>
>>> -
>>> define_resource(databaseCluster, 300, 0, 4, 0, ...)
>>>>>>                                                           -
>>>>>>
>>> update_resource_limits(databaseCluster, amrith, 1, 0, ...)
>>>>>>                                                           -
>>>>>>
>>> define_resource(storage, -1, 0, 20TB, 0, ...)
>>>>>>                                                           -
>>>>>>
>>> define_resource(memory, -1, 0, 10GB, 0, ...)
>>>>>>                                                           Assume
>>> that
>>>>>>                                                           the user
>>>>>>                                                           john
>>> comes
>>>>>>                                                           along and
>>>>>>                                                           asks for
>>> a
>>>>>>                                                           cluster
>>> with
>>>>>>                                                           4
>>>>>>                                                           nodes,
>>> 1TB
>>>>>>                                                           storage
>>> per
>>>>>>                                                           node and
>>>>>>                                                           each node
>>>>>>                                                           having
>>> 1GB
>>>>>>                                                           of
>>> memory,
>>>>>>                                                           the
>>>>>>                                                           system
>>> would
>>>>>>                                                           go
>>> through
>>>>>>                                                           the
>>>>>>                                                           following
>>>>>>                                                           sequence:
>>>>>>
>>>>>>                                                           -
>>>>>>
>>> reserve_resource(databaseCluster, john, 1, None)
>>>>>>                                                           o this
>>>>>>                                                           returns a
>>>>>>
>>> resourceID
>>>>>>                                                           (say
>>>>>>
>>> cluster-resource-
>>>>>>
>>>>>>                                   ID)
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                                                  o
>>> the
>>>>>>                                                           cluster
>>>>>>                                                           instance
>>>>>>                                                           that it
>>>>>>                                                           reserves
>>>>>>                                                           counts
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   against
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> the
>>>>>>                                                           limit of
>>> 300
>>>>>>                                                           cluster
>>>>>>                                                           instances
>>> in
>>>>>>                                                           total, as
>>>>>>                                                           well
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   as
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> the 4
>>>>>>                                                           clusters
>>>>>>                                                           that john
>>>>>>                                                           can
>>>>>>
>>> provision.
>>>>>>                                                           If
>>> 'amrith'
>>>>>>
>>>>>>
>>>>>>                                   had
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> requested
>>>>>>                                                           it, that
>>>>>>                                                           would
>>> have
>>>>>>                                                           been
>>> counted
>>>>>>                                                           against
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   the
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> limit
>>>>>>                                                           of 2
>>>>>>                                                           clusters
>>> for
>>>>>>                                                           the user.
>>>>>>
>>>>>>                                                           -
>>>>>>
>>> reserve_resource(databaseInstance, john, 1,
>>> cluster-resource-id)
>>>>>>                                                           -
>>>>>>
>>> reserve_resource(databaseInstance, john, 1,
>>> cluster-resource-id)
>>>>>>                                                           -
>>>>>>
>>> reserve_resource(databaseInstance, john, 1,
>>> cluster-resource-id)
>>>>>>                                                           -
>>>>>>
>>> reserve_resource(databaseInstance, john, 1,
>>> cluster-resource-id)
>>>>>>                                                           o this
>>>>>>                                                           returns
>>> four
>>>>>>                                                           resource
>>>>>>                                                           id's,
>>> let's
>>>>>>                                                           say
>>>>>>
>>> instance-1-id,  instance-2-id, instance-3-id,
>>> instance-4-id
>>>>>>                                                           o note
>>> that
>>>>>>                                                           each
>>>>>>                                                           instance
>>> is
>>>>>>                                                           that, an
>>>>>>                                                           instance
>>> by
>>>>>>                                                           itself.
>>> it
>>>>>>                                                           is
>>> therefore
>>>>>>                                                           not right
>>> to
>>>>>>                                                           consider
>>>>>>                                                           this
>>>>>>
>>>>>>
>>>>>>                                   as
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> equivalent
>>>>>>                                                           to a call
>>> to
>>> reserve_resource() with a
>>>>>>
>>>>>>
>>>>>>                                   size
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                                                  of
>>> 4,
>>> especially
>>>>>>                                                           because
>>> each
>>>>>>                                                           instance
>>>>>>                                                           could
>>> later
>>>>>>
>>>>>>
>>>>>>                                   be
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                                            tracked
>>> as
>>>>>>                                                           an
>>>>>>
>>> individual
>>>>>>                                                           Nova
>>>>>>                                                           instance.
>>>>>>
>>>>>>                                                           -
>>>>>>
>>> reserve_resource(storage, john, 1TB, instance-1-id)
>>>>>>                                                           -
>>>>>>
>>> reserve_resource(storage, john, 1TB, instance-2-id)
>>>>>>                                                           -
>>>>>>
>>> reserve_resource(storage, john, 1TB, instance-3-id)
>>>>>>                                                           -
>>>>>>
>>> reserve_resource(storage, john, 1TB, instance-4-id)
>>>>>>                                                           o each of
>>>>>>                                                           them
>>> returns
>>>>>>                                                           some
>>>>>>
>>> resourceID,
>>>>>>                                                           let's say
>>>>>>
>>>>>>
>>>>>>                                   they
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                                            returned
>>>>>>
>>> cinder-1-id,
>>> cinder-2-id,
>>> cinder-3-id,
>>> cinder-4-id
>>>>>>                                                           o since
>>> the
>>>>>>                                                           storage
>>> of
>>>>>>                                                           1TB is a
>>>>>>                                                           unit, it
>>> is
>>>>>>                                                           treated
>>>>>>
>>>>>>
>>>>>>                                   as
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> such.
>>>>>>                                                           In other
>>>>>>                                                           words,
>>> you
>>>>>>                                                           don't
>>> need
>>>>>>                                                           to invoke
>>>>>>
>>> reserve_resource 10^12 times, once per byte
>>>>>>                                                           allocated
>>>>>>                                                           :)
>>>>>>
>>>>>>                                                           -
>>>>>>
>>> reserve_resource(memory, john, 1GB, instance-1-id)
>>>>>>                                                           -
>>>>>>
>>> reserve_resource(memory, john, 1GB, instance-2-id)
>>>>>>                                                           -
>>>>>>
>>> reserve_resource(memory, john, 1GB, instance-3-id)
>>>>>>                                                           -
>>>>>>
>>> reserve_resource(memory, john, 1GB, instance-4-id)
>>>>>>                                                           o each of
>>>>>>                                                           these
>>> return
>>> something,
>>>>>>                                                           say
>>>>>>
>>> Dg4KBQcODAENBQEGBAcEDA, CgMJAg8FBQ8GDwgLBA8FAg,
>>> BAQJBwYMDwAIAA0DBAkNAg, AQMLDA4OAgEBCQ0MBAMGCA. I
>>>>>>
>>>>>>                                   have
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> made
>>>>>>                                                           up
>>> arbitrary
>>>>>>                                                           strings
>>> just
>>>>>>                                                           to
>>> highlight
>>>>>>                                                           that we
>>>>>>                                                           really
>>> don't
>>>>>>                                                           track
>>> these
>>>>>>                                                           anywhere
>>> so
>>>>>>                                                           we don't
>>>>>>                                                           care
>>>>>>
>>>>>>
>>>>>>                                                   about
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> them.
>>>>>>                                                           If all
>>> this
>>>>>>                                                           works,
>>> then
>>>>>>                                                           the
>>> system
>>>>>>                                                           knows
>>> that
>>>>>>                                                           John's
>>>>>>                                                           request
>>> does
>>>>>>                                                           not
>>> violate
>>>>>>                                                           any
>>> quotas
>>>>>>                                                           that it
>>> can
>>>>>>                                                           enforce,
>>> it
>>>>>>                                                           can then
>>> go
>>>>>>                                                           ahead and
>>>>>>                                                           launch
>>> the
>>>>>>                                                           instances
>>>>>>                                                           (calling
>>>>>>                                                           Nova),
>>>>>>                                                           provision
>>>>>>                                                           storage,
>>> and
>>>>>>                                                           so on.
>>>>>>
>>>>>>                                                           The
>>> system
>>>>>>                                                           then goes
>>>>>>                                                           and
>>> creates
>>>>>>                                                           four
>>> Cinder
>>>>>>                                                           volumes,
>>>>>>                                                           these are
>>>>>>
>>> cinder-1-uuid, cinder-2-uuid, cinder-3-uuid, cinder-4-uuid.
>>>>>>                                                           It can
>>> then
>>>>>>                                                           go and
>>>>>>                                                           confirm
>>>>>>                                                           those
>>>>>>
>>> reservations.
>>>>>>                                                           -
>>>>>>
>>> provision_resource(cinder-1-id, cinder-1-uuid)
>>>>>>                                                           -
>>>>>>
>>> provision_resource(cinder-2-id, cinder-2-uuid)
>>>>>>                                                           -
>>>>>>
>>> provision_resource(cinder-3-id, cinder-3-uuid)
>>>>>>                                                           -
>>>>>>
>>> provision_resource(cinder-4-id, cinder-4-uuid)
>>>>>>                                                           It could
>>>>>>                                                           then go
>>> and
>>>>>>                                                           launch 4
>>>>>>                                                           nova
>>>>>>                                                           instances
>>>>>>                                                           and
>>>>>>                                                           similarly
>>>>>>                                                           provision
>>>>>>                                                           those
>>>>>>
>>> resources,
>>>>>>                                                           and so
>>> on.
>>>>>>                                                           This
>>> process
>>>>>>                                                           could
>>> take
>>>>>>                                                           some
>>> minutes
>>>>>>                                                           and
>>>>>>                                                           holding a
>>>>>>                                                           database
>>>>>>
>>> transaction
>>>>>>                                                           open for
>>>>>>                                                           this is
>>> the
>>>>>>                                                           issue
>>> that
>>>>>>                                                           Jay
>>>>>>                                                           brings up
>>> in
>>>>>>                                                           [4]. We
>>>>>>                                                           don't
>>> have
>>>>>>                                                           to in
>>> this
>>>>>>                                                           proposed
>>>>>>                                                           scheme.
>>>>>>
>>>>>>                                                           Since the
>>>>>>                                                           resources
>>>>>>                                                           are all
>>>>>>
>>> hierarchically linked through the
>>>>>>                                                           overall
>>>>>>                                                           cluster
>>> id,
>>>>>>                                                           when the
>>>>>>                                                           cluster
>>> is
>>>>>>                                                           setup, it
>>>>>>                                                           can
>>> finally
>>>>>>                                                           go and
>>>>>>                                                           provision
>>>>>>                                                           that:
>>>>>>
>>>>>>                                                           -
>>>>>>
>>> provision_resource(cluster-resource-id, cluster-uuid)
>>>>>>                                                           When
>>> Trove
>>>>>>                                                           is done
>>> with
>>>>>>                                                           some
>>>>>>
>>> individual
>>>>>>                                                           resource,
>>> it
>>>>>>                                                           can go
>>> and
>>>>>>                                                           release
>>> it.
>>>>>>                                                           Note that
>>>>>>                                                           I'm
>>> thinking
>>>>>>                                                           this will
>>>>>>                                                           invoke
>>>>>>
>>> release_resource
>>>>>>                                                           with the
>>> ID
>>>>>>                                                           of the
>>>>>>
>>> underlying
>>>>>>                                                           object OR
>>>>>>                                                           the
>>>>>>                                                           resource.
>>>>>>
>>>>>>                                                           -
>>>>>>
>>> release_resource(cinder-4-id), and
>>>>>>                                                           -
>>>>>>
>>> release_resource(cinder-4-uuid)
>>>>>>                                                           are
>>>>>>                                                           therefore
>>>>>>                                                           identical
>>>>>>                                                           and
>>> indicate
>>>>>>                                                           that the
>>> 4th
>>>>>>                                                           1TB
>>> volume
>>>>>>                                                           is now
>>>>>>                                                           released.
>>>>>>                                                           How this
>>>>>>                                                           will be
>>>>>>
>>> implemented
>>>>>>                                                           in
>>> Python,
>>>>>>                                                           kwargs or
>>>>>>                                                           some
>>>>>>                                                           other
>>>>>>                                                           mechanism
>>>>>>                                                           is, I
>>>>>>                                                           believe,
>>> an
>>> implementation detail.
>>>>>>                                                           Finally,
>>> it
>>>>>>                                                           releases
>>> the
>>>>>>                                                           cluster
>>>>>>                                                           resource
>>> by
>>>>>>                                                           doing
>>> this:
>>>>>>                                                           -
>>>>>>
>>> release_resource(cluster-resource-id)
>>>>>>                                                           This
>>> would
>>>>>>                                                           release
>>> the
>>>>>>                                                           cluster
>>> and
>>>>>>                                                           all
>>>>>>                                                           dependent
>>>>>>                                                           resources
>>> in
>>>>>>                                                           a
>>>>>>                                                           single
>>>>>>
>>> operation.
>>>>>>                                                           A user
>>> may
>>>>>>                                                           wish to
>>>>>>                                                           manage a
>>>>>>                                                           resource
>>>>>>                                                           that was
>>>>>>
>>> provisioned
>>>>>>                                                           from the
>>>>>>                                                           service.
>>>>>>                                                           Assume
>>> that
>>>>>>                                                           this
>>> results
>>>>>>                                                           in a
>>>>>>                                                           resizing
>>> of
>>>>>>                                                           the
>>>>>>
>>> instances,
>>>>>>                                                           then it
>>> is a
>>>>>>                                                           matter of
>>>>>>                                                           updating
>>>>>>                                                           that
>>>>>>                                                           resource.
>>>>>>
>>>>>>                                                           Assume
>>> that
>>>>>>                                                           the third
>>>>>>                                                           1TB
>>> volume
>>>>>>                                                           is being
>>>>>>                                                           resized
>>> to
>>>>>>                                                           2TB, then
>>> it
>>>>>>                                                           is
>>>>>>                                                           merely a
>>>>>>                                                           matter of
>>>>>>                                                           invoking:
>>>>>>
>>>>>>                                                           -
>>>>>>
>>> update_resource(cinder-3-uuid, 2TB)
>>>>>>                                                           Delimiter
>>>>>>                                                           can go
>>>>>>                                                           figure
>>> out
>>>>>>                                                           that
>>>>>>
>>> cinder-3-uuid is a 1TB device and
>>>>>>                                                           therefore
>>>>>>                                                           this is
>>> an
>>>>>>                                                           increase
>>> of
>>>>>>                                                           1TB and
>>>>>>                                                           verify
>>> that
>>>>>>                                                           this is
>>>>>>                                                           within
>>>>>>                                                           the
>>> quotas
>>>>>>                                                           allowed
>>> for
>>>>>>                                                           the user.
>>>>>>
>>>>>>                                                           The thing
>>>>>>                                                           that I
>>> find
>>> attractive
>>>>>>                                                           about
>>> this
>>>>>>                                                           model of
>>>>>>
>>> maintaining
>>>>>>                                                           a
>>>>>>                                                           hierarchy
>>> of
>>> reservations
>>>>>>                                                           is that
>>> in
>>>>>>                                                           the event
>>> of
>>>>>>                                                           an error,
>>>>>>                                                           the
>>>>>>                                                           service
>>> need
>>>>>>                                                           merely
>>> call
>>> release_resource() on the highest level
>>> reservation
>>>>>>                                                           and the
>>>>>>                                                           Delimiter
>>>>>>                                                           project
>>> can
>>>>>>                                                           walk down
>>>>>>                                                           the chain
>>>>>>                                                           and
>>>>>>                                                           release
>>> all
>>>>>>                                                           the
>>>>>>                                                           resources
>>> or
>>> reservations
>>>>>>                                                           as
>>>>>>
>>> appropriate.
>>>>>>                                                           Under the
>>>>>>                                                           covers I
>>>>>>                                                           believe
>>> that
>>>>>>                                                           each of
>>>>>>                                                           these
>>>>>>
>>> operations
>>>>>>                                                           should be
>>>>>>                                                           atomic
>>> and
>>>>>>                                                           may
>>> update
>>>>>>                                                           multiple
>>>>>>                                                           database
>>>>>>                                                           tables
>>> but
>>>>>>                                                           these
>>> will
>>>>>>                                                           all be
>>>>>>                                                           short
>>> lived
>>> operations.
>>>>>>                                                           For
>>> example,
>>>>>>                                                           reserving
>>> an
>>>>>>                                                           instance
>>>>>>                                                           resource
>>>>>>                                                           would
>>>>>>                                                           increment
>>>>>>                                                           the
>>>>>>                                                           number of
>>>>>>                                                           instances
>>>>>>                                                           for the
>>> user
>>>>>>                                                           as well
>>> as
>>>>>>                                                           the
>>> number
>>>>>>                                                           of
>>> instances
>>>>>>                                                           on the
>>>>>>                                                           whole,
>>> and
>>>>>>                                                           this
>>> would
>>>>>>                                                           be an
>>> atomic
>>> operation.
>>>>>>                                                           I have
>>> two
>>>>>>                                                           primary
>>>>>>                                                           areas of
>>>>>>                                                           concern
>>>>>>                                                           about the
>>>>>>                                                           proposal
>>>>>>                                                           [3].
>>>>>>
>>>>>>                                                           The first
>>> is
>>>>>>                                                           that it
>>>>>>                                                           makes the
>>>>>>                                                           implicit
>>>>>>
>>> assumption
>>>>>>                                                           that the
>>>>>>                                                           "flat
>>> mode"
>>>>>>                                                           is
>>>>>>
>>> implemented.
>>>>>>                                                           That
>>>>>>                                                           provides
>>>>>>                                                           value to
>>> a
>>>>>>
>>>>>>                                   consumer
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                                            but I
>>> think
>>>>>>                                                           it leaves
>>> a
>>>>>>                                                           lot for
>>> the
>>>>>>                                                           consumer
>>> to
>>>>>>                                                           do. For
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                                   example,
>>>>>>
>>>>>>
>>>>>>
>>> I
>>>>>>                                                           find it
>>> hard
>>>>>>                                                           to see
>>> how
>>>>>>                                                           the model
>>>>>>                                                           proposed
>>>>>>                                                           would
>>> handle
>>>>>>
>>>>>>
>>>>>>                                   the
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                                            release
>>> of
>>>>>>                                                           quotas,
>>>>>>                                                           leave
>>> alone
>>>>>>                                                           the case
>>> of
>>>>>>                                                           a nested
>>>>>>                                                           release
>>> of
>>>>>>
>>>>>>                                                   a
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> hierarchy
>>>>>>                                                           of
>>>>>>
>>> resources.
>>>>>>                                                           The other
>>> is
>>>>>>                                                           the
>>> notion
>>>>>>                                                           that the
>>>>>>
>>> implementation will begin a
>>> transaction,
>>>>>>                                                           perform a
>>>>>>                                                           query(),
>>>>>>                                                           make some
>>>>>>
>>> manipulations, and
>>>>>>                                                           then do a
>>>>>>                                                           save().
>>> This
>>>>>>                                                           makes for
>>> an
>>> interesting
>>> transaction
>>> management
>>>>>>                                                           challenge
>>> as
>>>>>>                                                           it would
>>>>>>                                                           require
>>> the
>>> underlying
>>>>>>
>>>>>>                                                   database
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                                            to run
>>> in
>>>>>>                                                           an
>>> isolation
>>>>>>                                                           mode of
>>> at
>>>>>>                                                           least
>>>>>>
>>> repeatable
>>>>>>                                                           reads and
>>>>>>                                                           maybe
>>> even
>>> serializable
>>>>>>                                                           which
>>> would
>>>>>>                                                           be a
>>>>>>
>>> performance
>>>>>>                                                           bear on
>>>>>>
>>>>>>
>>>>>>                                   a
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                                            heavily
>>>>>>                                                           loaded
>>>>>>                                                           system.
>>> If
>>>>>>                                                           run in
>>> the
>>> traditional
>>>>>>                                                           read-
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                   committed
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                                            mode,
>>> this
>>>>>>                                                           would
>>>>>>                                                           silently
>>>>>>                                                           lead to
>>> over
>>> subscriptions, and
>>>>>>
>>>>>>
>>>>>>                                   the
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> violation
>>>>>>                                                           of quota
>>>>>>                                                           limits.
>>>>>>
>>>>>>                                                           I believe
>>>>>>                                                           that it
>>>>>>                                                           should be
>>> a
>>> requirement
>>>>>>                                                           that the
>>>>>>                                                           Delimiter
>>>>>>                                                           library
>>>>>>                                                           should be
>>>>>>                                                           able to
>>> run
>>>>>>                                                           against a
>>>>>>                                                           database
>>>>>>                                                           that
>>>>>>                                                           supports,
>>>>>>                                                           and is
>>>>>>
>>> configured
>>>>>>                                                           for
>>>>>>
>>> READ-COMMITTED, and should not require anything higher.
>>>>>>                                                           The model
>>>>>>                                                           proposed
>>>>>>                                                           above can
>>>>>>                                                           certainly
>>> be
>>> implemented
>>>>>>                                                           with a
>>>>>>                                                           database
>>>>>>                                                           running
>>>>>>
>>> READ-COMMITTED, and I believe that this is also
>>>>>>                                                           true with
>>>>>>                                                           the
>>> caveat
>>>>>>                                                           that the
>>>>>>
>>> operations
>>>>>>                                                           will be
>>>>>>                                                           performed
>>>>>>                                                           through
>>>>>>
>>>>>>
>>>>>>                                   SQLAlchemy.
>>>>>>
>>>>>>
>>>>>>                                                           Thanks,
>>>>>>
>>>>>>                                                           -amrith
>>>>>>
>>>>>>                                                           [1]
>>>>>>
>>> http://openstack.markmail.org/thread/tkl2jcyvzgifniux
>>>>>>                                                           [2]
>>>>>>
>>> http://openstack.markmail.org/thread/3cr7hoeqjmgyle2j
>>>>>>                                                           [3]
>>>>>>
>>> https://review.openstack.org/#/c/284454/
>>>>>>                                                           [4]
>>>>>>
>>> http://markmail.org/message/7ixvezcsj3uyiro6
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> ____________________________________________________________________
>>>>>>                                                           __ ____
>>>>>>                                                           OpenStack
>>>>>>
>>> Development
>>>>>>                                                           Mailing
>>> List
>>>>>>                                                           (not for
>>>>>>                                                           usage
>>>>>>
>>> questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>> _____________________________________________________________________
>>>>>>                                                   _____ OpenStack
>>>>>>                                                   Development
>>> Mailing
>>>>>>                                                   List (not for
>>> usage
>>>>>>                                                   questions)
>>>>>>                                                   Unsubscribe:
>>>>>>
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>> ______________________________________________________________________
>>>>>>                                           ____ OpenStack
>>> Development
>>>>>>                                           Mailing List (not for
>>> usage
>>>>>>                                           questions)
>>>>>>                                           Unsubscribe:
>>>>>>
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>> __________________________________________________________________________
>>>
>>>>>>                                   OpenStack Development Mailing
>>> List
>>>>>>                                   (not for usage questions)
>>>>>>                                   Unsubscribe:
>>>>>>
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>>>>>
>>> __________________________________________________________________________
>>>
>>>>>>                           OpenStack Development Mailing List (not
>>> for
>>>>>>                           usage questions)
>>>>>>                           Unsubscribe:
>>>>>>
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>>>>>
>>> __________________________________________________________________________
>>>
>>>>>>                   OpenStack Development Mailing List (not for usage
>>>>>>                   questions)
>>>>>>                   Unsubscribe:
>>>>>>
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> __________________________________________________________________________
>>>
>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>
>>>>>> Unsubscribe:
>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>
>>> __________________________________________________________________________
>>>
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> __________________________________________________________________________
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



More information about the OpenStack-dev mailing list