[openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal
Jay Pipes
jaypipes at gmail.com
Tue Apr 26 16:33:27 UTC 2016
On 04/25/2016 02:05 PM, Joshua Harlow wrote:
> Is the generation stuff going to be exposed outside of the AP?
No, wasn't planning on it.
> I'm sort of hoping not(?), because a service (if one everyone wanted to
> create it) for say zookeeper (or etcd or consul...) could use its
> built-in generation equivalent (every znode has a version that u can use
> to do equivalent things).
>
> Thus it's like the gist I posted earlier @
>
> https://gist.github.com/harlowja/e7175c2d76e020a82ae94467a1441d85
>
> So might be nice to not expose such a thing outside of the db-layer.
>
> Amrith Kumar wrote:
>> On Sat, 2016-04-23 at 21:41 +0000, Amrith Kumar wrote:
>>> Ok to beer and high bandwidth. FYI Jay the distributed high perf db we
>>> did a couple of years ago is now open source. Just saying. Mysql plug
>>> compatible ....
>>
>>> -amrith
>>>
>>>
>>> --
>>> Amrith Kumar
>>> amrith at tesora.com
>>>
>>>
>>> -------- Original message --------
>>> From: Jay Pipes<jaypipes at gmail.com>
>>> Date: 04/23/2016 4:10 PM (GMT-05:00)
>>> To: Amrith Kumar<amrith at tesora.com>,
>>> openstack-dev at lists.openstack.org
>>> Cc: vilobhmm at yahoo-inc.com, nik.komawar at gmail.com, Ed Leafe
>>> <ed at leafe.com>
>>> Subject: Re: [openstack-dev] More on the topic of DELIMITER, the Quota
>>> Management Library proposal
>>>
>>>
>>> Looking forward to arriving in Austin so that I can buy you a beer,
>>> Amrith, and have a high-bandwidth conversation about how you're
>>> wrong. :P
>>
>>
>> Jay and I chatted and it took a long time to come to an agreement
>> because we weren't able to find any beer.
>>
>> Here's what I think we've agreed about. The library will store data in
>> two tables;
>>
>> 1. the detail table which stores the individual claims and the resource
>> class.
>> 2. a generation table which stores the resource class and a generation.
>>
>> When a claim is received, the requestor performs the following
>> operations.
>>
>> begin
>>
>> select sum(detail.claims) as total_claims,
>> generation.resource as resource,
>> generation.generation last_generation
>> from detail, generation
>> where detail.resource = generation.resource
>> and generation.resource =<chosen resource; memory, cpu, ...>
>> group by generation.generation, generation.resource
>>
>> if total_claims + this_claim< limit
>> insert into detail values (this_claim, resource)
>>
>> update generation
>> set generation = generation + 1
>> where generation = last_generation
>>
>> if @@rowcount = 1
>> -- all good
>> commit
>> else
>> rollback
>> -- try again
>>
>>
>>
>> There will be some bootstrapping that will be required for the situation
>> where there are no detail records for a given resource and so on but I
>> think we can figure that out easily. The easiest way I can think of
>> doing that is to lose the join and do both the queries (one against the
>> detail and one against the generation table within the same
>> transaction).
>>
>> Using the generation table update as the locking mechanism that prevents
>> multiple requestors from making concurrent claims.
>>
>> So long as people don't go and try and read these tables and change
>> tables outside of the methods that the library provides, we can
>> guarantee that this is al safe and will not oversubscribe.
>>
>> -amrith
>>
>>> Comments inline.
>>>
>>> On 04/23/2016 11:25 AM, Amrith Kumar wrote:
>>>> On Sat, 2016-04-23 at 10:26 -0400, Andrew Laski wrote:
>>>>> On Fri, Apr 22, 2016, at 09:57 PM, Tim Bell wrote:
>>>>>
>>>>>> I have reservations on f and g.
>>>>>>
>>>>>>
>>>>>> On f., We have had a number of discussions in the past about
>>>>>> centralising quota (e.g. Boson) and the project teams of the other
>>>>>> components wanted to keep the quota contents ‘close’. This can
>>>>>> always be reviewed further with them but I would hope for at least
>>> a
>>>>>> standard schema structure of tables in each project for the
>>> handling
>>>>>> of quota.
>>>>>>
>>>>>>
>>>>>> On g., aren’t all projects now nested projects ? If we have the
>>>>>> complexity of handling nested projects sorted out in the common
>>>>>> library, is there a reason why a project would not want to support
>>>>>> nested projects ?
>>>>>>
>>>>>>
>>>>>> One other issue is how to do reconcilliation, each project needs
>>> to
>>>>>> have a mechanism to re-calculate the current allocations and
>>>>>> reconcile that with the quota usage. While in an ideal world, this
>>>>>> should not be necessary, it would be for the foreseeable future,
>>>>>> especially with a new implementation.
>>>>>>
>>>>> One of the big reasons that Jay and I have been pushing to remove
>>>>> reservations and tracking of quota in a separate place than the
>>>>> resources are actually used, e.g., an instance record in the Nova
>>> db,
>>>>> is so that reconciliation is not necessary. For example, if RAM
>>> quota
>>>>> usage is simply tracked as sum(instances.memory_mb) then you can be
>>>>> sure that usage is always up to date.
>>>> Uh oh, there be gremlins here ...
>>>>
>>>> I am positive that this will NOT work, see earlier conversations
>>> about
>>>> isolation levels, and Jay's alternate solution.
>>>>
>>>> The way (I understand the issue, and Jay's solution) you get around
>>> the
>>>> isolation levels trap is to NOT do your quota determinations based
>>> on a
>>>> SUM(column) but rather based on the rowcount on a well crafted
>>> UPDATE of
>>>> a single table that stored total quota.
>>> No, we would do our quota calculations by doing a SUM(used) against
>>> the
>>> allocations table. There is no separate table that stored the total
>>> quota (or quota usage records). That's the source of the problem with
>>> the existing quota handling code in Nova. The generation field value
>>> is
>>> used to provide the consistent view of the actual resource usage
>>> records
>>> so that the INSERT operations for all claimed resources can be done in
>>> a
>>> transactional manner and will be rolled back if any other writer
>>> changes
>>> the amount of consumed resources on a provider (which of course would
>>> affect the quota check calculations).
>>>
>>> > You could also store a detail
>>>> claim record for each claim in an independent table that is
>>> maintained
>>>> in the same database transaction if you so desire, that is optional.
>>> The allocations table is the "detail claim record" table that you
>>> refer
>>> to above.
>>>
>>>> My view of how this would work (which I described earlier as
>>> building on
>>>> Jay's solution) is that the claim flow would look like this:
>>>>
>>>> select total_used, generation
>>>> from quota_claimed
>>>> where tenant =<tenant> and resource = 'memory'
>>> There is no need to keep a total_used value for anything. That is
>>> denormalized calculated data that merely adds a point of race
>>> contention. The quota check is against the *detail* table
>>> (allocations),
>>> which stores the *actual resource usage records*.
>>>
>>>> begin transaction
>>>>
>>>> update quota_claimed
>>>> set total_used = total_used + claim, generation =
>>>> generation + 1
>>>> where tenant =<tenant> and resource = 'memory'
>>>> and generation = generation
>>>> and total_used + claim< limit
>>> This part of the transaction must always occur **after** the
>>> insertion
>>> of the actual resource records, not before.
>>>
>>>> if @@rowcount = 1
>>>> -- optional claim_detail table
>>>> insert into claim_detail values (<tenant>,
>>> 'memory',
>>>> claim, ...)
>>>> commit
>>>> else
>>>> rollback
>>> So, in pseudo-Python-SQLish code, my solution works like this:
>>>
>>> limits = get_limits_from_delimiter()
>>> requested = get_requested_from_request_spec()
>>>
>>> while True:
>>>
>>> used := SELECT
>>> resource_class,
>>> resource_provider,
>>> generation,
>>> SUM(used) as total_used
>>> FROM allocations
>>> JOIN resource_providers ON (...)
>>> WHERE consumer_uuid = $USER_UUID
>>> GROUP BY
>>> resource_class,
>>> resource_provider,
>>> generation;
>>>
>>> # Check that our requested resource amounts don't exceed quotas
>>> if not check_requested_within_limits(requested, used, limits):
>>> raise QuotaExceeded
>>>
>>> # Claim all requested resources. Note that the generation
>>> retrieved
>>> # from the above query is our consistent view marker. If the
>>> UPDATE
>>> # below succeeds and returns != 0 rows affected, that means there
>>> # was no other writer that changed our resource usage in between
>>> # this thread's claiming of resources, and therefore we prevent
>>> # any oversubscription of resources.
>>> begin_transaction:
>>>
>>> provider := SELECT id, generation, ... FROM
>>> resource_providers
>>> JOIN (...)
>>> WHERE (<resource_usage_filters>)
>>>
>>> for resource in requested:
>>> INSERT INTO allocations (
>>> resource_provider_id,
>>> resource_class_id,
>>> consumer_uuid,
>>> used
>>> ) VALUES (
>>> $provider.id,
>>> $resource.id,
>>> $USER_UUID,
>>> $resource.amount
>>> );
>>>
>>> rows_affected := UPDATE resource_providers
>>> SET generation = generation + 1
>>> WHERE id = $provider.id
>>> AND generation =
>>> $used[$provider.id].generation;
>>>
>>> if $rows_affected == 0:
>>> ROLLBACK;
>>>
>>> The only reason we would need a post-claim quota check is if some of
>>> the
>>> requested resources are owned and tracked by an external-to-Nova
>>> system.
>>>
>>> BTW, note to Ed Leafe... unless your distributed data store supports
>>> transactional semantics, you can't use a distributed data store for
>>> these types of solutions. Instead, you will need to write a whole
>>> bunch
>>> of code that does post-auditing of claims and quotas and a system
>>> that
>>> accepts that oversubscription and out-of-sync quota limits and usages
>>> is
>>> a fact of life. Not to mention needing to implement JOINs in Python.
>>>
>>>> But, it is my understanding that
>>>>
>>>> (a) if you wish to do the SUM(column) approach that you
>>> propose,
>>>> you must have a reservation that is committed and then you
>>> must
>>>> re-read the SUM(column) to make sure you did not
>>> over-subscribe;
>>>> and
>>> Erm, kind of? Oversubscription is not possible in the solution I
>>> describe because the compare-and-update on the
>>> resource_providers.generation field allows for a consistent view of
>>> the
>>> resources used -- and if that view changes during the insertion of
>>> resource usage records -- the transaction containing those insertions
>>> is
>>> rolled back.
>>>
>>>> (b) to get away from reservations you must stop using the
>>>> SUM(column) approach and instead use a single quota_claimed
>>>> table to determine the current quota claimed.
>>> No. This has nothing to do with reservations.
>>>
>>>> At least that's what I understand of Jay's example from earlier in
>>> this
>>>> thread.
>>>>
>>>> Let's definitely discuss this in Austin. While I don't love Jay's
>>>> solution for other reasons to do with making the quota table a
>>> hotspot
>>>> and things like that, it is a perfectly workable solution, I think.
>>> There is no quota table in my solution.
>>>
>>> If you refer to the resource_providers table (the table that has the
>>> generation field), then yes, it's a hot spot. But hot spots in the DB
>>> aren't necessarily a bad thing if you design the underlying schema
>>> properly.
>>>
>>> More in Austin.
>>>
>>> Best,
>>> -jay
>>>
>>>>>
>>>>>> Tim
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> From: Amrith Kumar<amrith at tesora.com>
>>>>>> Reply-To: "OpenStack Development Mailing List (not for usage
>>>>>> questions)"<openstack-dev at lists.openstack.org>
>>>>>> Date: Friday 22 April 2016 at 06:51
>>>>>> To: "OpenStack Development Mailing List (not for usage questions)"
>>>>>> <openstack-dev at lists.openstack.org>
>>>>>> Subject: Re: [openstack-dev] More on the topic of DELIMITER, the
>>>>>> Quota Management Library proposal
>>>>>>
>>>>>>
>>>>>>
>>>>>> I’ve thought more about Jay’s approach to enforcing
>>> quotas
>>>>>> and I think we can build on and around it. With that
>>>>>> implementation as the basic quota primitive, I think we
>>> can
>>>>>> build a quota management API that isn’t dependent on
>>>>>> reservations. It does place some burdens on the consuming
>>>>>> projects that I had hoped to avoid and these will cause
>>>>>> heartburn for some (make sure that you always request
>>>>>> resources in a consistent order and free them in a
>>>>>> consistent order being the most obvious).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> If it doesn’t make it harder, I would like to see if we
>>> can
>>>>>> make the quota API take care of the ordering of requests.
>>>>>> i.e. if the quota API is an extension of Jay’s example
>>> and
>>>>>> accepts some data structure (dict?) with all the claims
>>> that
>>>>>> a project wants to make for some operation, and then
>>>>>> proceeds to make those claims for the project in the
>>>>>> consistent order, I think it would be of some value.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Beyond that, I’m on board with a-g below,
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -amrith
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> From: Vilobh Meshram
>>>>>> [mailto:vilobhmeshram.openstack at gmail.com]
>>>>>> Sent: Friday, April 22, 2016 4:08 AM
>>>>>> To: OpenStack Development Mailing List (not for usage
>>>>>> questions)<openstack-dev at lists.openstack.org>
>>>>>> Subject: Re: [openstack-dev] More on the topic of
>>> DELIMITER,
>>>>>> the Quota Management Library proposal
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> I strongly agree with Jay on the points related to "no
>>>>>> reservation" , keeping the interface simple and the role
>>> for
>>>>>> Delimiter (impose limits on resource consumption and
>>> enforce
>>>>>> quotas).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> The point to keep user quota, tenant quotas in Keystone
>>>>>> sounds interestring and would need support from Keystone
>>>>>> team. We have a Cross project session planned [1] and
>>> will
>>>>>> definitely bring that up in that session.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> The main thought with which Delimiter was formed was to
>>>>>> enforce resource quota in transaction safe manner and do
>>> it
>>>>>> in a cross-project conducive manner and it still holds
>>>>>> true. Delimiters mission is to impose limits on
>>>>>> resource consumption and enforce quotas in transaction
>>> safe
>>>>>> manner. Few key aspects of Delimiter are :-
>>>>>>
>>>>>>
>>>>>>
>>>>>> a. Delimiter will be a new Library and not a Service.
>>>>>> Details covered in spec.
>>>>>>
>>>>>>
>>>>>> b. Delimiter's role will be to impose limits on resource
>>>>>> consumption.
>>>>>>
>>>>>>
>>>>>> c. Delimiter will not be responsible for rate limiting.
>>>>>>
>>>>>>
>>>>>> d. Delimiter will not maintain data for the resources.
>>>>>> Respective projects will take care of keeping,
>>> maintaining
>>>>>> data for the resources and resource consumption.
>>>>>>
>>>>>>
>>>>>> e. Delimiter will not have the concept of "reservations".
>>>>>> Delimiter will read or update the "actual" resource
>>> tables
>>>>>> and will not rely on the "cached" tables. At present, the
>>>>>> quota infrastructure in Nova, Cinder and other projects
>>> have
>>>>>> tables such as reservations, quota_usage, etc which are
>>> used
>>>>>> as "cached tables" to track re
>>>>>>
>>>>>>
>>>>>> f. Delimiter will fetch the information for project
>>> quota,
>>>>>> user quota from a centralized place, say Keystone, or if
>>>>>> that doesn't materialize will fetch default quota values
>>>>>> from respective service. This information will be cached
>>>>>> since it gets updated rarely but read many times.
>>>>>>
>>>>>>
>>>>>> g. Delimiter will take into consideration whether the
>>>>>> project is a Flat or Nested and will make the
>>> calculations
>>>>>> of allocated, available resources. Nested means project
>>>>>> namespace is hierarchical and Flat means project
>>> namespace
>>>>>> is not hierarchical.
>>>>>>
>>>>>>
>>>>>> -Vilobh
>>>>>>
>>>>>>
>>>>>> [1]
>>> https://www.openstack.org/summit/austin-2016/summit-schedule/events/9492
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Apr 21, 2016 at 11:08 PM, Joshua Harlow
>>>>>> <harlowja at fastmail.com> wrote:
>>>>>>
>>>>>>
>>>>>> Since people will be on a plane soon,
>>>>>>
>>>>>> I threw this together as a example of a quota
>>> engine
>>>>>> (the zookeeper code does even work, and yes it
>>>>>> provides transactional semantics due to the nice
>>>>>> abilities of zookeeper znode versions[1] and its
>>>>>> inherent consistency model, yippe).
>>>>>>
>>>>>>
>>> https://gist.github.com/harlowja/e7175c2d76e020a82ae94467a1441d85
>>>>>> Someone else can fill in the db quota engine with
>>> a
>>>>>> similar/equivalent api if they so dare, ha. Or
>>> even
>>>>>> feel to say the gist/api above is crap, cause
>>> that's
>>>>>> ok to, lol.
>>>>>>
>>>>>> [1]
>>>>>>
>>> https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#Data
>>> +Access
>>>>>>
>>>>>>
>>>>>> Amrith Kumar wrote:
>>>>>>
>>>>>> Inline below ... thread is too long, will
>>>>>> catch you in Austin.
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Jay Pipes
>>>>>> [mailto:jaypipes at gmail.com]
>>>>>> Sent: Thursday, April 21, 2016
>>> 8:08
>>>>>> PM
>>>>>> To:
>>>>>> openstack-dev at lists.openstack.org
>>>>>> Subject: Re: [openstack-dev] More
>>> on
>>>>>> the topic of DELIMITER, the Quota
>>>>>> Management Library proposal
>>>>>>
>>>>>> Hmm, where do I start... I think
>>> I
>>>>>> will just cut to the two primary
>>>>>> disagreements I have. And I will
>>>>>> top-post because this email is
>>> way
>>>>>> too
>>>>>> big.
>>>>>>
>>>>>> 1) On serializable isolation
>>> level.
>>>>>> No, you don't need it at all to
>>>>>> prevent races in claiming. Just
>>> use
>>>>>> a
>>>>>> compare-and-update with retries
>>>>>> strategy. Proof is here:
>>>>>>
>>>>>>
>>> https://github.com/jaypipes/placement-bench/blob/master/placement.py#L97-
>>>
>>>>>> L142
>>>>>>
>>>>>> Works great and prevents multiple
>>>>>> writers from oversubscribing any
>>>>>> resource without relying on any
>>>>>> particular isolation level at
>>> all.
>>>>>> The `generation` field in the
>>>>>> inventories table is what allows
>>>>>> multiple
>>>>>> writers to ensure a consistent
>>> view
>>>>>> of the data without needing to
>>> rely
>>>>>> on
>>>>>> heavy lock-based semantics and/or
>>>>>> RDBMS-specific isolation levels.
>>>>>>
>>>>>>
>>>>>>
>>>>>> [amrith] this works for what it is doing,
>>> we
>>>>>> can definitely do this. This will work at
>>>>>> any isolation level, yes. I didn't want
>>> to
>>>>>> go this route because it is going to
>>> still
>>>>>> require an insert into another table
>>>>>> recording what the actual 'thing' is that
>>> is
>>>>>> claiming the resource and that insert is
>>>>>> going to be in a different transaction
>>> and
>>>>>> managing those two transactions was what
>>> I
>>>>>> wanted to avoid. I was hoping to avoid
>>>>>> having two tables tracking claims, one
>>>>>> showing the currently claimed quota and
>>>>>> another holding the things that claimed
>>> that
>>>>>> quota. Have to think again whether that
>>> is
>>>>>> possible.
>>>>>>
>>>>>> 2) On reservations.
>>>>>>
>>>>>> The reason I don't believe
>>>>>> reservations are necessary to be
>>> in
>>>>>> a quota
>>>>>> library is because reservations
>>> add
>>>>>> a concept of a time to a claim of
>>>>>> some
>>>>>> resource. You reserve some
>>> resource
>>>>>> to be claimed at some point in
>>> the
>>>>>> future and release those
>>> resources
>>>>>> at a point further in time.
>>>>>>
>>>>>> Quota checking doesn't look at
>>> what
>>>>>> the state of some system will be
>>> at
>>>>>> some point in the future. It
>>> simply
>>>>>> returns whether the system *right
>>>>>> now* can handle a request *right
>>>>>> now* to claim a set of resources.
>>>>>>
>>>>>> If you want reservation semantics
>>>>>> for some resource, that's totally
>>>>>> cool,
>>>>>> but IMHO, a reservation service
>>>>>> should live outside of the
>>> service
>>>>>> that is
>>>>>> actually responsible for
>>> providing
>>>>>> resources to a consumer.
>>>>>> Merging right-now quota checks
>>> and
>>>>>> future-based reservations into
>>> the
>>>>>> same
>>>>>> library just complicates things
>>>>>> unnecessarily IMHO.
>>>>>>
>>>>>>
>>>>>>
>>>>>> [amrith] extension of the above ...
>>>>>>
>>>>>> 3) On resizes.
>>>>>>
>>>>>> Look, I recognize some users see
>>>>>> some value in resizing their
>>>>>> resources.
>>>>>> That's fine. I personally think
>>>>>> expand operations are fine, and
>>> that
>>>>>> shrink operations are really the
>>>>>> operations that should be
>>> prohibited
>>>>>> in
>>>>>> the API. But, whatever, I'm fine
>>>>>> with resizing of requested
>>> resource
>>>>>> amounts. My big point is if you
>>>>>> don't have a separate table that
>>>>>> stores
>>>>>> quota_usages and instead only
>>> have a
>>>>>> single table that stores the
>>> actual
>>>>>> resource usage records, you don't
>>>>>> have to do *any* quota check
>>>>>> operations
>>>>>> at all upon deletion of a
>>> resource.
>>>>>> For modifying resource amounts
>>> (i.e.
>>>>>> a
>>>>>> resize) you merely need to change
>>>>>> the calculation of requested
>>>>>> resource
>>>>>> amounts to account for the
>>>>>> already-consumed usage amount.
>>>>>>
>>>>>> Bottom line for me: I really
>>> won't
>>>>>> support any proposal for a
>>> complex
>>>>>> library that takes the resource
>>>>>> claim process out of the hands of
>>>>>> the
>>>>>> services that own those
>>> resources.
>>>>>> The simpler the interface of this
>>>>>> library, the better.
>>>>>>
>>>>>>
>>>>>>
>>>>>> [amrith] my proposal would not but this
>>>>>> email thread has got too long. Yes,
>>> simpler
>>>>>> interface, will catch you in Austin.
>>>>>>
>>>>>> Best,
>>>>>> -jay
>>>>>>
>>>>>> On 04/19/2016 09:59 PM, Amrith
>>> Kumar
>>>>>> wrote:
>>>>>>
>>>>>> -----Original
>>>>>> Message-----
>>>>>> From: Jay Pipes
>>>>>>
>>> [mailto:jaypipes at gmail.com]
>>>>>> Sent: Monday,
>>> April
>>>>>> 18, 2016 2:54 PM
>>>>>> To:
>>>>>>
>>> openstack-dev at lists.openstack.org
>>>>>> Subject: Re:
>>>>>> [openstack-dev]
>>> More
>>>>>> on the topic of
>>>>>> DELIMITER, the
>>>>>> Quota Management
>>>>>> Library proposal
>>>>>>
>>>>>> On 04/16/2016
>>> 05:51
>>>>>> PM, Amrith Kumar
>>>>>> wrote:
>>>>>>
>>>>>> If we
>>>>>> therefore
>>>>>> assume
>>> that
>>>>>> this will
>>> be
>>>>>> a Quota
>>>>>>
>>> Management
>>>>>> Library,
>>>>>> it is
>>> safe
>>>>>> to assume
>>>>>> that
>>> quotas
>>>>>> are going
>>> to
>>>>>> be
>>> managed
>>>>>> on a
>>>>>>
>>> per-project
>>>>>> basis,
>>> where
>>> participating projects will use this library.
>>>>>> I believe
>>>>>> that it
>>>>>> stands to
>>>>>> reason
>>> that
>>>>>> any data
>>>>>>
>>> persistence
>>>>>> will
>>>>>> have to
>>> be
>>>>>> in a
>>>>>> location
>>>>>> decided
>>> by
>>>>>> the
>>>>>>
>>> individual
>>>>>> project.
>>>>>>
>>>>>>
>>>>>> Depends on what
>>> you
>>>>>> mean by "any data
>>>>>> persistence". If
>>> you
>>>>>> are
>>>>>> referring to the
>>>>>> storage of quota
>>>>>> values (per user,
>>>>>> per tenant,
>>>>>> global, etc) I
>>> think
>>>>>> that should be
>>> done
>>>>>> by the Keystone
>>>>>> service.
>>>>>> This data is
>>>>>> essentially an
>>>>>> attribute of the
>>>>>> user or the
>>> tenant
>>>>>> or the
>>>>>>
>>>>>>
>>>>>> service endpoint itself (i.e.
>>>>>>
>>>>>>
>>>>>> global defaults).
>>>>>> This data also
>>>>>> rarely changes
>>> and
>>>>>> logically belongs
>>>>>> to the service
>>> that
>>>>>> manages users,
>>>>>> tenants, and
>>> service
>>>>>> endpoints:
>>>>>>
>>>>>>
>>>>>> Keystone.
>>>>>>
>>>>>>
>>>>>> If you are
>>> referring
>>>>>> to the storage of
>>>>>> resource usage
>>>>>> records, yes,
>>>>>> each service
>>> project
>>>>>> should own that
>>> data
>>>>>> (and frankly, I
>>>>>> don't see a
>>>>>> need to persist
>>> any
>>>>>> quota usage data
>>> at
>>>>>> all, as I
>>> mentioned
>>>>>> in a
>>>>>> previous reply to
>>>>>> Attila).
>>>>>>
>>>>>>
>>>>>> [amrith] You make a
>>>>>> distinction that I had
>>> made
>>>>>> implicitly, and it is
>>>>>> important to highlight
>>> it.
>>>>>> Thanks for pointing it
>>> out.
>>>>>> Yes, I meant
>>>>>> both of the above, and as
>>>>>> stipulated. Global
>>> defaults
>>>>>> in keystone
>>>>>> (somehow, TBD) and usage
>>>>>> records, on a per-service
>>>>>> basis.
>>>>>>
>>>>>> That may
>>> not
>>>>>> be a very
>>>>>>
>>> interesting
>>>>>> statement
>>>>>> but the
>>>>>> corollary
>>>>>> is, I
>>>>>> think, a
>>>>>> very
>>>>>>
>>> significant
>>> statement;
>>>>>> it cannot
>>> be
>>>>>> assumed
>>> that
>>>>>> the
>>>>>> quota
>>>>>>
>>> management
>>> information
>>>>>> for all
>>>>>>
>>> participating projects is in
>>>>>> the same
>>>>>> database.
>>>>>>
>>>>>>
>>>>>> It cannot be
>>> assumed
>>>>>> that this
>>>>>> information is
>>> even
>>>>>> in a database at
>>>>>>
>>>>>>
>>>>>>
>>>>>> all...
>>>>>>
>>>>>>
>>>>>> [amrith] I don't follow.
>>> If
>>>>>> the service in question
>>> is
>>>>>> to be scalable,
>>>>>> I think it stands to
>>> reason
>>>>>> that there must be some
>>>>>> mechanism by which
>>>>>> instances of the service
>>> can
>>>>>> share usage records (as
>>> you
>>>>>> refer to
>>>>>> them, and I like that
>>> term).
>>>>>> I think it stands to
>>> reason
>>>>>> that there
>>>>>> must be some database,
>>> no?
>>>>>> A
>>>>>>
>>> hypothetical
>>>>>> service
>>>>>> consuming
>>>>>> the
>>>>>> Delimiter
>>>>>> library
>>>>>> provides
>>>>>>
>>> requesters
>>>>>> with some
>>>>>> widgets,
>>> and
>>>>>> wishes to
>>>>>> track the
>>>>>> widgets
>>> that
>>>>>> it has
>>>>>>
>>> provisioned
>>>>>> both on a
>>>>>> per-user
>>>>>> basis,
>>> and
>>>>>> on the
>>>>>> whole. It
>>>>>> should
>>>>>> therefore
>>>>>>
>>> multi-tenant
>>>>>> and able
>>> to
>>>>>> track the
>>>>>> widgets
>>> on a
>>>>>> per
>>>>>> tenant
>>> basis
>>>>>> and if
>>>>>> required
>>>>>> impose
>>>>>> limits on
>>>>>> the
>>> number
>>>>>> of
>>> widgets
>>>>>> that a
>>>>>> tenant
>>> may
>>>>>> consume
>>> at a
>>>>>> time,
>>> during
>>>>>> a course
>>> of
>>>>>> a period
>>> of
>>>>>> time, and
>>> so
>>>>>> on.
>>>>>>
>>>>>>
>>>>>> No, this last
>>> part
>>>>>> is absolutely not
>>>>>> what I think
>>> quota
>>>>>> management
>>>>>> should be about.
>>>>>>
>>>>>> Rate limiting --
>>>>>> i.e. how many
>>>>>> requests a
>>>>>> particular user
>>> can
>>>>>> make of
>>>>>> an API in a given
>>>>>> period of time --
>>>>>> should *not* be
>>>>>> handled by
>>>>>> OpenStack API
>>>>>> services, IMHO.
>>> It
>>>>>> is the
>>>>>> responsibility of
>>>>>> the
>>>>>> deployer to
>>> handle
>>>>>> this using
>>>>>> off-the-shelf
>>>>>> rate-limiting
>>>>>> solutions
>>>>>>
>>>>>>
>>>>>> (open source or proprietary).
>>>>>>
>>>>>>
>>>>>> Quotas should
>>> only
>>>>>> be about the hard
>>>>>> limit of
>>> different
>>>>>> types of
>>>>>> resources that a
>>>>>> user or group of
>>>>>> users can consume
>>> at
>>>>>> a given time.
>>>>>>
>>>>>>
>>>>>> [amrith] OK, good point.
>>>>>> Agreed as stipulated.
>>>>>>
>>>>>>
>>>>>> Such a
>>>>>>
>>> hypothetical
>>>>>> service
>>> may
>>>>>> also
>>> consume
>>>>>> resources
>>>>>> from
>>> other
>>>>>> services
>>>>>> that it
>>>>>> wishes to
>>>>>> track,
>>> and
>>>>>> impose
>>>>>> limits
>>> on.
>>>>>>
>>>>>> Yes, absolutely
>>>>>> agreed.
>>>>>>
>>>>>>
>>>>>> It is
>>> also
>>> understood
>>>>>> as Jay
>>> Pipes
>>>>>> points
>>> out
>>>>>> in [4]
>>> that
>>>>>> the
>>> actual
>>>>>> process
>>> of
>>> provisioning
>>>>>> widgets
>>>>>> could be
>>>>>> time
>>>>>> consuming
>>>>>> and it is
>>>>>>
>>> ill-advised
>>>>>> to hold a
>>>>>> database
>>>>>>
>>> transaction
>>>>>> of any
>>> kind
>>>>>> open for
>>>>>> that
>>>>>> duration
>>> of
>>>>>> time.
>>>>>> Ensuring
>>>>>> that a
>>> user
>>>>>> does not
>>>>>> exceed
>>> some
>>>>>> limit on
>>>>>> the
>>> number
>>>>>> of
>>>>>>
>>> concurrent
>>>>>> widgets
>>> that
>>>>>> he or she
>>>>>> may
>>> create
>>>>>> therefore
>>>>>> requires
>>>>>> some
>>>>>> mechanism
>>> to
>>>>>> track
>>>>>> in-flight
>>>>>> requests
>>> for
>>>>>> widgets.
>>> I
>>>>>> view
>>> these
>>>>>> as
>>> "intent"
>>>>>> but not
>>> yet
>>> materialized.
>>>>>>
>>>>>> It has nothing to
>>> do
>>>>>> with the amount
>>> of
>>>>>> concurrent
>>> widgets
>>>>>> that a
>>>>>> user can create.
>>>>>> It's just about
>>> the
>>>>>> total number of
>>> some
>>>>>> resource
>>>>>> that may be
>>> consumed
>>>>>> by that user.
>>>>>>
>>>>>> As for an
>>> "intent",
>>>>>> I don't believe
>>>>>> tracking intent
>>> is
>>>>>> the right way
>>>>>> to go at all. As
>>>>>> I've mentioned
>>>>>> before, the major
>>>>>> problem in Nova's
>>>>>> quota system is
>>> that
>>>>>> there are two
>>> tables
>>>>>> storing resource
>>>>>> usage
>>>>>> records: the
>>>>>> *actual* resource
>>>>>> usage tables (the
>>>>>> allocations table
>>> in
>>>>>> the new
>>>>>> resource-
>>> providers
>>>>>> modeling and the
>>>>>> instance_extra,
>>>>>> pci_devices and
>>>>>> instances table
>>> in
>>>>>> the legacy
>>> modeling)
>>>>>> and the *quota
>>>>>> usage* tables
>>>>>> (quota_usages and
>>>>>> reservations
>>>>>> tables). The
>>>>>> quota_usages
>>> table
>>>>>> does
>>>>>> not need to exist
>>> at
>>>>>> all, and neither
>>>>>> does the
>>>>>> reservations
>>> table.
>>>>>> Don't do
>>>>>> intent-based
>>>>>> consumption.
>>>>>> Instead, just
>>>>>> consume (claim)
>>> by
>>>>>> writing a record
>>> for
>>>>>> the resource
>>> class
>>>>>> consumed on a
>>>>>> provider into
>>>>>> the actual
>>> resource
>>>>>> usages table and
>>>>>> then "check
>>> quotas"
>>>>>> by querying
>>>>>> the *actual*
>>>>>> resource usages
>>> and
>>>>>> comparing the
>>>>>> SUM(used) values,
>>>>>> grouped by
>>> resource
>>>>>> class, against
>>> the
>>>>>> appropriate quota
>>>>>> limits for
>>>>>> the user. The
>>>>>> introduction of
>>> the
>>>>>> quota_usages and
>>>>>> reservations
>>>>>> tables to cache
>>>>>> usage records is
>>> the
>>>>>> primary reason
>>> for
>>>>>> the race
>>>>>> problems in the
>>> Nova
>>>>>> (and
>>>>>> other) quota
>>> system
>>>>>> because every
>>> time
>>>>>> you introduce a
>>>>>> caching system
>>>>>> for
>>> highly-volatile
>>>>>> data (like usage
>>>>>> records) you
>>>>>> introduce
>>>>>> complexity into
>>> the
>>>>>> write path and
>>> the
>>>>>> need to track the
>>>>>> same thing
>>>>>> across multiple
>>>>>> writes to
>>> different
>>>>>> tables
>>> needlessly.
>>>>>>
>>>>>> [amrith] I don't agree,
>>> I'll
>>>>>> respond to this and the
>>> next
>>>>>> comment group
>>>>>>
>>>>>>
>>>>>>
>>>>>> together. See below.
>>>>>>
>>>>>>
>>>>>> Looking
>>> up
>>>>>> at this
>>>>>> whole
>>>>>>
>>> infrastructure from the perspective of the
>>>>>> database,
>>> I
>>>>>> think we
>>>>>> should
>>>>>> require
>>> that
>>>>>> the
>>> database
>>>>>> must not
>>> be
>>>>>> required
>>> to
>>>>>> operate
>>> in
>>>>>> any
>>>>>> isolation
>>>>>> mode
>>> higher
>>>>>> than
>>>>>>
>>> READ-COMMITTED; more about that later (i.e. requiring a database run
>>>>>> either
>>>>>>
>>> serializable
>>>>>> or
>>>>>>
>>> repeatable
>>>>>> read is a
>>>>>> show
>>>>>> stopper).
>>>>>>
>>>>>>
>>>>>> This is an
>>>>>> implementation
>>>>>> detail is not
>>>>>> relevant to the
>>>>>> discussion
>>>>>> about what the
>>>>>> interface of a
>>> quota
>>>>>> library would
>>> look
>>>>>> like.
>>>>>>
>>>>>>
>>>>>> [amrith] I disagree, let
>>> me
>>>>>> give you an example of
>>> why.
>>>>>> Earlier, I wrote:
>>>>>>
>>>>>> Such a
>>>>>>
>>> hypothetical
>>>>>> service
>>> may
>>>>>> also
>>> consume
>>>>>> resources
>>>>>> from
>>> other
>>>>>> services
>>>>>> that it
>>>>>> wishes to
>>>>>> track,
>>> and
>>>>>> impose
>>>>>> limits
>>> on.
>>>>>>
>>>>>> And you responded:
>>>>>>
>>>>>>
>>>>>> Yes, absolutely
>>>>>> agreed.
>>>>>>
>>>>>>
>>>>>>
>>>>>> So let's take this
>>>>>> hypothetical service that
>>> in
>>>>>> response to a user
>>>>>>
>>>>>>
>>>>>>
>>>>>> request, will provision a Cinder
>>>>>> volume and a Nova instance. Let's
>>>>>> assume
>>>>>> that the service also imposes
>>> limits
>>>>>> on the number of cinder volumes
>>> and
>>>>>> nova instances the user may
>>>>>> provision; independent of limits
>>>>>> that Nova and
>>>>>> Cinder may themselves maintain.
>>>>>>
>>>>>> One way that the
>>>>>> hypothetical service can
>>>>>> function is this:
>>>>>>
>>>>>> (a) check Cinder quota,
>>> if
>>>>>> successful, create cinder
>>>>>> volume
>>>>>> (b) check Nova quota, if
>>>>>> successful, create nova
>>>>>> instance with cinder
>>>>>> volume attachment
>>>>>>
>>>>>> Now, this is sub-optimal
>>> as
>>>>>> there are going to be
>>> some
>>>>>> number of cases
>>>>>>
>>>>>>
>>>>>> where the nova quota check fails.
>>>>>> Now you have needlessly created
>>> and
>>>>>> will
>>>>>> have to release a cinder volume.
>>> It
>>>>>> also takes longer to fail.
>>>>>>
>>>>>> Another way to do this is
>>>>>> this:
>>>>>>
>>>>>> (1) check Cinder quota,
>>> if
>>>>>> successful, check Nova
>>>>>> quota, if successful
>>>>>> proceed to (2) else error
>>>>>> out
>>>>>> (2) create cinder volume
>>>>>> (3) create nova instance
>>>>>> with cinder attachment.
>>>>>>
>>>>>> I'm trying to get to this
>>>>>> latter form of doing
>>> things.
>>>>>> Easy, you might say ...
>>>>>> theoretically this should
>>>>>> simply be:
>>>>>>
>>>>>> BEGIN;
>>>>>> -- Get data to do the
>>> Cinder
>>>>>> check
>>>>>>
>>>>>> SELECT ......
>>>>>>
>>>>>> -- Do the cinder check
>>>>>>
>>>>>> INSERT INTO ....
>>>>>>
>>>>>> -- Get data to do the
>>> Nova
>>>>>> check
>>>>>>
>>>>>> SELECT ....
>>>>>>
>>>>>> -- Do the Nova check
>>>>>>
>>>>>> INSERT INTO ...
>>>>>>
>>>>>> COMMIT
>>>>>>
>>>>>> You can only make this
>>> work
>>>>>> if you ran at isolation
>>>>>> level serializable.
>>>>>>
>>>>>>
>>>>>> Why?
>>>>>>
>>>>>>
>>>>>> To make this run at
>>>>>> isolation level
>>>>>> REPEATABLE-READ, you must
>>>>>> enforce
>>>>>>
>>>>>>
>>>>>>
>>>>>> constraints at the database level
>>>>>> that will fail the commit. But
>>> wait,
>>>>>> you
>>>>>> can't do that because the data
>>> about
>>>>>> the global limits may not be in
>>> the
>>>>>> same database as the usage
>>> records.
>>>>>> Later you talk about caching and
>>>>>> stuff; all that doesn't help a
>>>>>> database constraint.
>>>>>>
>>>>>> For this reason, I think
>>>>>> there is going to have to
>>> be
>>>>>> some cognizance to
>>>>>>
>>>>>>
>>>>>>
>>>>>> the database isolation level in
>>> the
>>>>>> design of the library, and I
>>> think
>>>>>> it
>>>>>> will also impact the API that can
>>> be
>>>>>> constructed.
>>>>>>
>>>>>> In
>>> general
>>> therefore, I
>>>>>> believe
>>> that
>>>>>> the
>>>>>>
>>> hypothetical
>>>>>> service
>>>>>>
>>> processing
>>>>>> requests
>>> for
>>>>>> widgets
>>>>>> would
>>> have
>>>>>> to handle
>>>>>> three
>>> kinds
>>>>>> of
>>>>>>
>>> operations,
>>> provision,
>>>>>> modify,
>>> and
>>>>>> destroy.
>>> The
>>>>>> names
>>> are, I
>>>>>> believe,
>>>>>>
>>> self-explanatory.
>>>>>>
>>>>>> Generally,
>>>>>> modification of a
>>>>>> resource doesn't
>>>>>> come into play.
>>> The
>>>>>> primary exception
>>> to
>>>>>> this is for
>>>>>> transferring of
>>>>>> ownership of some
>>>>>>
>>>>>>
>>>>>> resource.
>>>>>>
>>>>>>
>>>>>> [amrith] Trove RESIZE is
>>> a
>>>>>> huge benefit for users
>>> and
>>>>>> while it may be a
>>>>>>
>>>>>>
>>>>>>
>>>>>> pain as you say, this is still a
>>>>>> very real benefit. Trove allows
>>> you
>>>>>> to
>>>>>> resize both your storage (resize
>>> the
>>>>>> cinder volume) and resize your
>>>>>> instance (change the flavor).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Without
>>> loss
>>>>>> of
>>>>>>
>>> generality,
>>>>>> one can
>>> say
>>>>>> that all
>>>>>> three of
>>>>>> them must
>>>>>> validate
>>>>>> that the
>>>>>> operation
>>>>>> does not
>>>>>> violate
>>> some
>>>>>> limit (no
>>>>>> more
>>>>>> than X
>>>>>> widgets,
>>> no
>>>>>> fewer
>>> than X
>>>>>> widgets,
>>>>>> rates,
>>> and
>>>>>> so on).
>>>>>>
>>>>>>
>>>>>> No, only the
>>>>>> creation (and
>>> very
>>>>>> rarely the
>>>>>> modification)
>>> needs
>>>>>> any
>>>>>> validation that a
>>>>>> limit could been
>>>>>> violated.
>>> Destroying
>>>>>> a resource
>>>>>> never needs to be
>>>>>> checked for limit
>>>>>> violations.
>>>>>>
>>>>>>
>>>>>> [amrith] Well, if you are
>>>>>> going to create a volume
>>> of
>>>>>> 10GB and your
>>>>>>
>>>>>>
>>>>>>
>>>>>> limit is 100GB, resizing it to
>>> 200GB
>>>>>> should fail, I think.
>>>>>>
>>>>>>
>>>>>> Assuming
>>>>>> that the
>>>>>> service
>>>>>>
>>> provisions
>>>>>> resources
>>>>>> from
>>> other
>>>>>> services,
>>>>>> it is
>>> also
>>> conceivable
>>>>>> that
>>> limits
>>>>>> be
>>> imposed
>>>>>> on the
>>>>>> quantum
>>> of
>>>>>> those
>>>>>> services
>>>>>> consumed.
>>> In
>>>>>> practice,
>>> I
>>>>>> can
>>> imagine
>>>>>> a service
>>>>>> like
>>>>>> Trove
>>> using
>>>>>> the
>>>>>> Delimiter
>>>>>> project
>>> to
>>>>>> perform
>>> all
>>>>>> of these
>>>>>> kinds of
>>>>>> limit
>>>>>> checks;
>>> I'm
>>>>>> not
>>>>>>
>>> suggesting
>>>>>> that it
>>> does
>>>>>> this
>>> today,
>>>>>> nor that
>>>>>> there is
>>> an
>>>>>> immediate
>>>>>> plan to
>>>>>> implement
>>>>>> all of
>>> them,
>>>>>> just that
>>>>>> these
>>>>>> all seem
>>>>>> like good
>>>>>> uses a
>>> Quota
>>> Management
>>> capability.
>>>>>> - User
>>> may
>>>>>> not have
>>>>>> more than
>>> 25
>>>>>> database
>>>>>> instances
>>> at
>>>>>> a
>>>>>>
>>>>>>
>>>>>> time
>>>>>>
>>>>>>
>>>>>>
>>> -
>>>>>> User may
>>> not
>>>>>> have more
>>>>>> than 4
>>>>>> clusters
>>> at
>>>>>> a time
>>>>>> - User
>>> may
>>>>>> not
>>> consume
>>>>>> more than
>>>>>> 3TB of
>>> SSD
>>>>>> storage
>>> at a
>>>>>> time
>>>>>>
>>>>>>
>>>>>> Only if SSD
>>> storage
>>>>>> is a distinct
>>>>>> resource class
>>> from
>>>>>> DISK_GB. Right
>>>>>> now, Nova makes
>>> no
>>>>>> differentiation
>>>>>> w.r.t. SSD or HDD
>>> or
>>>>>> shared vs.
>>>>>> local block
>>> storage.
>>>>>>
>>>>>> [amrith] It matters not
>>> to
>>>>>> Trove whether Nova does
>>> nor
>>>>>> not. Cinder
>>>>>>
>>>>>>
>>>>>>
>>>>>> supports volume-types and users
>>> DO
>>>>>> want to limit based on
>>> volume-type
>>>>>> (for
>>>>>> example).
>>>>>>
>>>>>>
>>> -
>>>>>> User may
>>> not
>>>>>> launch
>>> more
>>>>>> than 10
>>> huge
>>>>>> instances
>>> at
>>>>>> a
>>>>>> time
>>>>>>
>>>>>>
>>>>>> What is the point
>>> of
>>>>>> such a limit?
>>>>>>
>>>>>>
>>>>>>
>>>>>> [amrith] Metering usage,
>>>>>> placing limitations on
>>> the
>>>>>> quantum of resources
>>>>>>
>>>>>>
>>>>>>
>>>>>> that a user may provision. Same
>>> as
>>>>>> with Nova. A flavor is merely a
>>>>>> simple
>>>>>> way to tie together a bag of
>>>>>> resources. It is a way to
>>> restrict
>>>>>> access,
>>>>>> for example, to specific
>>> resources
>>>>>> that are available in the cloud.
>>>>>> HUGE
>>>>>> is just an example I gave, pick
>>> any
>>>>>> flavor you want, and here's how a
>>>>>> service like Trove uses it.
>>>>>>
>>>>>> Users can ask to launch
>>> an
>>>>>> instance of a specific
>>>>>> database+version;
>>>>>>
>>>>>>
>>>>>>
>>>>>> MySQL 5.6-48 for example. Now, an
>>>>>> operator can restrict the
>>> instance
>>>>>> flavors, or volume types that can
>>> be
>>>>>> associated with the specific
>>>>>> datastore. And the flavor could
>>> be
>>>>>> used to map to, for example
>>> whether
>>>>>> the
>>>>>> instance is running on bare metal
>>> or
>>>>>> in a VM and if so with what kind
>>> of
>>>>>> hardware. That's a useful
>>> construct
>>>>>> for a service like Trove.
>>>>>>
>>>>>>
>>> -
>>>>>> User may
>>> not
>>>>>> launch
>>> more
>>>>>> than 3
>>>>>> clusters
>>> an
>>>>>> hour
>>>>>>
>>>>>>
>>>>>>
>>>>>> -1. This is rate
>>>>>> limiting and
>>> should
>>>>>> be handled by
>>>>>> rate-limiting
>>>>>>
>>>>>>
>>>>>>
>>>>>> services.
>>>>>>
>>>>>>
>>>>>>
>>> -
>>>>>> No more
>>> than
>>>>>> 500
>>> copies
>>>>>> of Oracle
>>>>>> may be
>>> run
>>>>>> at a time
>>>>>>
>>>>>>
>>>>>>
>>>>>> Is "Oracle" a
>>>>>> resource class?
>>>>>>
>>>>>>
>>>>>>
>>>>>> [amrith] As I view it,
>>> every
>>>>>> project should be free to
>>>>>> define its own
>>>>>>
>>>>>>
>>>>>>
>>>>>> set of resource classes and meter
>>>>>> them as it feels fit. So, while
>>>>>> Oracle
>>>>>> licenses may not, conceivably a
>>> lot
>>>>>> of things that Nova, Cinder, and
>>> the
>>>>>> other core projects don't care
>>>>>> about, are in fact relevant for a
>>>>>> consumer
>>>>>> of this library.
>>>>>>
>>>>>> While
>>> Nova
>>>>>> would be
>>> the
>>>>>> service
>>> that
>>>>>> limits
>>> the
>>>>>> number of
>>>>>> instances
>>>>>> a user
>>> can
>>>>>> have at a
>>>>>> time, the
>>>>>> ability
>>> for
>>>>>> a service
>>> to
>>>>>> limit
>>> this
>>>>>> further
>>>>>> should
>>> not
>>>>>> be
>>>>>>
>>> underestimated.
>>>>>> In turn,
>>>>>> should
>>> Nova
>>>>>> and
>>> Cinder
>>>>>> also use
>>> the
>>>>>> same
>>> Quota
>>> Management
>>>>>> Library,
>>>>>> they may
>>>>>> each
>>> impose
>>> limitations
>>>>>> like:
>>>>>>
>>>>>> - User
>>> may
>>>>>> not
>>> launch
>>>>>> more than
>>> 20
>>>>>> huge
>>>>>> instances
>>> at
>>>>>> a
>>>>>> time
>>>>>>
>>>>>>
>>>>>> Not a useful
>>>>>> limitation IMHO.
>>>>>>
>>>>>>
>>>>>>
>>>>>> [amrith] I beg to differ.
>>>>>> Again a huge instance is
>>>>>> just an example of
>>>>>>
>>>>>>
>>>>>>
>>>>>> some flavor; and the idea is to
>>>>>> allow a project to place its own
>>>>>> metrics
>>>>>> and meter based on those.
>>>>>>
>>>>>>
>>> -
>>>>>> User may
>>> not
>>>>>> launch
>>> more
>>>>>> than 3
>>>>>> instances
>>> in
>>>>>> a minute
>>>>>>
>>>>>>
>>>>>>
>>>>>> -1. This is rate
>>>>>> limiting.
>>>>>>
>>>>>>
>>>>>>
>>> -
>>>>>> User may
>>> not
>>>>>> consume
>>> more
>>>>>> than 15TB
>>> of
>>>>>> SSD at a
>>>>>> time
>>>>>> - User
>>> may
>>>>>> not have
>>>>>> more than
>>> 30
>>>>>> volumes
>>> at a
>>>>>> time
>>>>>>
>>>>>> Again,
>>> I'm
>>>>>> not
>>> implying
>>>>>> that
>>> either
>>>>>> Nova or
>>>>>> Cinder
>>>>>> should
>>>>>> provide
>>>>>> these
>>>>>>
>>> capabilities.
>>>>>> With this
>>> in
>>>>>> mind, I
>>>>>> believe
>>> that
>>>>>> the
>>> minimal
>>>>>> set of
>>>>>>
>>> operations
>>>>>> that
>>>>>> Delimiter
>>>>>> should
>>>>>> provide
>>> are:
>>>>>> -
>>>>>>
>>> define_resource(name, max, min, user_max, user_min, ...)
>>>>>>
>>>>>> What would the
>>> above
>>>>>> do? What service
>>>>>> would it be
>>> speaking
>>>>>> to?
>>>>>>
>>>>>>
>>>>>>
>>>>>> [amrith] I assume that
>>> this
>>>>>> would speak with some
>>>>>> backend (either
>>>>>>
>>>>>>
>>>>>>
>>>>>> keystone or the project itself)
>>> and
>>>>>> record these designated limits.
>>> This
>>>>>> is the way to register a project
>>>>>> specific metric like "Oracle
>>>>>> licenses".
>>>>>>
>>>>>>
>>> -
>>> update_resource_limits(name, user, user_max, user_min,
>>>>>> ...)
>>>>>>
>>>>>>
>>>>>> This doesn't
>>> belong
>>>>>> in a quota
>>> library.
>>>>>> It belongs as a
>>> REST
>>>>>> API in
>>>>>> Keystone.
>>>>>>
>>>>>>
>>>>>> [amrith] Fine, same place
>>>>>> where the previous thing
>>>>>> stores the global
>>>>>>
>>>>>>
>>>>>>
>>>>>> defaults is the target of this
>>> call.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> -
>>> reserve_resource(name, user, size, parent_resource, ...)
>>>>>>
>>>>>>
>>>>>> This doesn't
>>> belong
>>>>>> in a quota
>>> library
>>>>>> at all. I think
>>>>>> reservations
>>>>>> are not germane
>>> to
>>>>>> resource
>>> consumption
>>>>>> and should be
>>>>>> handled by an
>>>>>> external service
>>> at
>>>>>> the orchestration
>>>>>> layer.
>>>>>>
>>>>>>
>>>>>> [amrith] Again not true,
>>> as
>>>>>> illustrated above this
>>>>>> library is the thing
>>>>>>
>>>>>>
>>>>>>
>>>>>> that projects could use to
>>> determine
>>>>>> whether or not to honor a
>>> request.
>>>>>> This reserve/provision process
>>> is, I
>>>>>> believe required because of the
>>>>>> vagaries of how we want to
>>> implement
>>>>>> this in the database.
>>>>>>
>>>>>>
>>> -
>>> provision_resource(resource, id)
>>>>>>
>>>>>>
>>>>>> A quota library
>>>>>> should not be
>>>>>> provisioning
>>>>>> anything. A quota
>>>>>> library
>>>>>> should simply
>>>>>> provide a
>>> consistent
>>>>>> interface for
>>>>>> *checking* that a
>>>>>> structured
>>> request
>>>>>> for some set of
>>>>>> resources *can*
>>> be
>>>>>> provided by the
>>>>>> service.
>>>>>>
>>>>>>
>>>>>> [amrith] This does not
>>>>>> actually call Nova or
>>>>>> anything; merely that
>>> AFTER
>>>>>>
>>>>>>
>>>>>> the hypothetical service has
>>> called
>>>>>> NOVA, this converts the
>>> reservation
>>>>>> (which can expire) into an actual
>>>>>> allocation.
>>>>>>
>>>>>>
>>> -
>>> update_resource(id or resource, newsize)
>>>>>>
>>>>>>
>>>>>> Resizing
>>> resources
>>>>>> is a bad idea,
>>> IMHO.
>>>>>> Resources are
>>> easier
>>>>>> to deal
>>>>>> with when they
>>> are
>>>>>> considered of
>>>>>> immutable size
>>> and
>>>>>> simple (i.e. not
>>>>>> complex or
>>> nested).
>>>>>> I think the
>>> problem
>>>>>> here is in the
>>>>>> definition of
>>>>>> resource classes
>>>>>> improperly.
>>>>>>
>>>>>>
>>>>>> [amrith] Let's leave the
>>>>>> quota library aside. This
>>>>>> assertion strikes at
>>>>>>
>>>>>>
>>>>>>
>>>>>> the very heart of things like
>>> Nova
>>>>>> resize, or for that matter Cinder
>>>>>> volume resize. Are those all bad
>>>>>> ideas? I made a 500GB Cinder
>>> volume
>>>>>> and
>>>>>> it is getting close to full. I'd
>>>>>> like to resize it to 2TB. Are you
>>>>>> saying
>>>>>> that's not a valid use case?
>>>>>>
>>>>>> For example, a
>>>>>> "cluster" is not
>>> a
>>>>>> resource. It is a
>>>>>> collection of
>>>>>> resources of type
>>>>>> node. "Resizing"
>>> a
>>>>>> cluster is a
>>>>>> misnomer, because
>>>>>> you aren't
>>> resizing
>>>>>> a resource at
>>> all.
>>>>>> Instead, you are
>>>>>> creating or
>>>>>> destroying
>>> resources
>>>>>> inside the
>>> cluster
>>>>>> (i.e. joining or
>>>>>> leaving
>>>>>>
>>>>>>
>>>>>> cluster nodes).
>>>>>>
>>>>>>
>>>>>> BTW, this is also
>>>>>> why the "resize
>>>>>> instance" API in
>>>>>> Nova is such a
>>>>>> giant pain in the
>>>>>> ass. It's
>>> attempting
>>>>>> to "modify" the
>>>>>> instance
>>>>>>
>>>>>>
>>>>>> "resource"
>>>>>>
>>>>>>
>>>>>> when the instance
>>>>>> isn't really the
>>>>>> resource at all.
>>> The
>>>>>> VCPU, RAM_MB,
>>>>>> DISK_GB, and PCI
>>>>>> devices are the
>>>>>> actual resources.
>>>>>> The instance is a
>>>>>> convenient way to
>>>>>> tie those
>>> resources
>>>>>> together, and
>>> doing
>>>>>> a "resize"
>>>>>> of the instance
>>>>>> behind the scenes
>>>>>> actually performs
>>> a
>>>>>> *move*
>>>>>> operation, which
>>>>>> isn't a *change*
>>> of
>>>>>> the original
>>>>>> resources.
>>> Rather,
>>>>>> it is a creation
>>> of
>>>>>> a new set of
>>>>>> resources (of the
>>>>>> new amounts) and
>>> a
>>>>>> deletion of the
>>> old
>>>>>> set of resources.
>>>>>>
>>>>>>
>>>>>> [amrith] that's fine, if
>>> all
>>>>>> we want is to handle the
>>>>>> resize operation
>>>>>>
>>>>>>
>>>>>>
>>>>>> as a new instance followed by a
>>>>>> deletion, that's great. But that
>>>>>> semantic
>>>>>> isn't necessarily the case for
>>>>>> something like (say) cinder.
>>>>>>
>>>>>> The "resize" API
>>>>>> call adds some
>>> nasty
>>>>>> confirmation and
>>>>>> cancel
>>>>>> semantics to the
>>>>>> calling interface
>>>>>> that hint that
>>> the
>>>>>> underlying
>>>>>> implementation of
>>>>>> the "resize"
>>>>>> operation is in
>>>>>> actuality not a
>>>>>> resize
>>>>>> at all, but
>>> rather a
>>> create-new-and-delete-old-resources operation.
>>>>>>
>>>>>> [amrith] And that isn't
>>>>>> germane to a quota
>>> library,
>>>>>> I don't think. What
>>>>>>
>>>>>>
>>>>>>
>>>>>> is, is this. Do we want to treat
>>> the
>>>>>> transient state when there are
>>> (for
>>>>>> example of Nova) two instances,
>>> one
>>>>>> of the new flavor and one of the
>>> old
>>>>>> flavor, or not. But, from the
>>>>>> perspective of a quota library, a
>>>>>> resize
>>>>>> operation is merely a reset of
>>> the
>>>>>> quota by the delta in the
>>> resource
>>>>>> consumed.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> -
>>> release_resource(id or resource)
>>>>>> -
>>>>>>
>>> expire_reservations()
>>>>>>
>>>>>> I see no need to
>>>>>> have reservations
>>> in
>>>>>> the quota library
>>> at
>>>>>> all, as
>>>>>> mentioned above.
>>>>>>
>>>>>>
>>>>>> [amrith] Then I think the
>>>>>> quota library must
>>> require
>>>>>> that either (a) the
>>>>>>
>>>>>>
>>>>>>
>>>>>> underlying database runs
>>>>>> serializable or (b) database
>>>>>> constraints can be
>>>>>> used to enforce that at commit
>>> the
>>>>>> global limits are adhered to.
>>>>>>
>>>>>> As for your
>>> proposed
>>>>>> interface and
>>>>>> calling structure
>>>>>> below, I think a
>>>>>> much simpler
>>>>>> proposal would
>>> work
>>>>>> better. I'll work
>>> on
>>>>>> a cross-project
>>>>>> spec that
>>> describes
>>>>>> this simpler
>>>>>> proposal, but the
>>>>>> basics would be:
>>>>>>
>>>>>> 1) Have Keystone
>>>>>> store quota
>>>>>> information for
>>>>>> defaults (per
>>>>>> service
>>>>>> endpoint), for
>>>>>> tenants and for
>>>>>> users.
>>>>>>
>>>>>> Keystone would
>>> have
>>>>>> the set of
>>> canonical
>>>>>> resource class
>>>>>> names, and
>>>>>> each project,
>>> upon
>>>>>> handling a new
>>>>>> resource class,
>>>>>> would be
>>>>>> responsible for a
>>>>>> change submitted
>>> to
>>>>>> Keystone to add
>>> the
>>>>>> new resource
>>>>>>
>>>>>>
>>>>>> class code.
>>>>>>
>>>>>>
>>>>>> Straw man REST
>>> API:
>>>>>>
>>> GET /quotas/resource-classes
>>>>>> 200 OK
>>>>>> {
>>>>>>
>>> "resource_classes":
>>>>>> {
>>>>>> "compute.vcpu": {
>>>>>> "service":
>>>>>> "compute",
>>>>>> "code":
>>>>>> "compute.vcpu",
>>>>>> "description": "A
>>>>>> virtual CPU unit"
>>>>>> },
>>>>>> "compute.ram_mb":
>>> {
>>>>>> "service":
>>>>>> "compute",
>>>>>> "code":
>>>>>> "compute.ram_mb",
>>>>>> "description":
>>>>>> "Memory in
>>>>>> megabytes"
>>>>>> },
>>>>>> ...
>>>>>> "volume.disk_gb":
>>> {
>>>>>> "service":
>>> "volume",
>>>>>> "code":
>>>>>> "volume.disk_gb",
>>>>>> "description":
>>>>>> "Amount of disk
>>>>>> space in
>>> gigabytes"
>>>>>> },
>>>>>> ...
>>>>>> "database.count":
>>> {
>>>>>> "service":
>>>>>> "database",
>>>>>> "code":
>>>>>> "database.count",
>>>>>> "description":
>>>>>> "Number of
>>> database
>>>>>> instances"
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>>
>>>>>>
>>>>>> [amrith] Well, a user is
>>>>>> allowed to have a certain
>>>>>> compute quota (which
>>>>>>
>>>>>>
>>>>>>
>>>>>> is shared by Nova and Trove) but
>>>>>> also a Trove quota. How would
>>> your
>>>>>> representation represent that?
>>>>>>
>>>>>> # Get the default
>>>>>> limits for new
>>>>>> users...
>>>>>>
>>> GET /quotas/defaults
>>>>>> 200 OK
>>>>>> {
>>>>>> "quotas": {
>>>>>> "compute.vcpu":
>>> 100,
>>>>>> "compute.ram_mb":
>>>>>> 32768,
>>>>>> "volume.disk_gb":
>>>>>> 1000,
>>>>>> "database.count":
>>> 25
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> # Get a specific
>>>>>> user's limits...
>>>>>>
>>> GET /quotas/users/{UUID}
>>>>>> 200 OK
>>>>>> {
>>>>>> "quotas": {
>>>>>> "compute.vcpu":
>>> 100,
>>>>>> "compute.ram_mb":
>>>>>> 32768,
>>>>>> "volume.disk_gb":
>>>>>> 1000,
>>>>>> "database.count":
>>> 25
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> # Get a tenant's
>>>>>> limits...
>>>>>>
>>> GET /quotas/tenants/{UUID}
>>>>>> 200 OK
>>>>>> {
>>>>>> "quotas": {
>>>>>> "compute.vcpu":
>>>>>> 1000,
>>>>>> "compute.ram_mb":
>>>>>> 327680,
>>>>>> "volume.disk_gb":
>>>>>> 10000,
>>>>>> "database.count":
>>>>>> 250
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> 2) Have Delimiter
>>>>>> communicate with
>>> the
>>>>>> above proposed
>>> new
>>>>>> Keystone
>>>>>> REST API and
>>> package
>>>>>> up data into an
>>>>>>
>>> oslo.versioned_objects interface.
>>>>>> Clearly all of
>>> the
>>>>>> above can be
>>> heavily
>>>>>> cached both on
>>> the
>>>>>> server and
>>>>>> client side since
>>>>>> they rarely
>>> change
>>>>>> but are read
>>> often.
>>>>>>
>>>>>> [amrith] Caching on the
>>>>>> client won't save you
>>> from
>>>>>> oversubscription if
>>>>>>
>>>>>>
>>>>>>
>>>>>> you don't run serializable.
>>>>>>
>>>>>>
>>>>>> The Delimiter
>>>>>> library could be
>>>>>> used to provide a
>>>>>> calling interface
>>>>>> for service
>>> projects
>>>>>> to get a user's
>>>>>> limits for a set
>>> of
>>>>>> resource
>>>>>>
>>>>>>
>>>>>> classes:
>>>>>>
>>>>>>
>>>>>> (please excuse
>>>>>> wrongness, typos,
>>>>>> and other stuff
>>>>>> below, it's just
>>> a
>>>>>> straw- man not
>>>>>> production
>>> working
>>>>>> code...)
>>>>>>
>>>>>> # file:
>>>>>>
>>> delimiter/objects/limits.py
>>>>>> import
>>>>>>
>>> oslo.versioned_objects.base as ovo import
>>> oslo.versioned_objects.fields as ovo_fields
>>>>>>
>>>>>> class
>>>>>>
>>> ResourceLimit(ovo.VersionedObjectBase):
>>>>>> # 1.0: Initial
>>>>>> version
>>>>>> VERSION = '1.0'
>>>>>>
>>>>>> fields = {
>>>>>> 'resource_class':
>>>>>>
>>> ovo_fields.StringField(),
>>>>>> 'amount':
>>>>>>
>>> ovo_fields.IntegerField(),
>>>>>> }
>>>>>>
>>>>>>
>>>>>> class
>>>>>>
>>> ResourceLimitList(ovo.VersionedObjectBase):
>>>>>> # 1.0: Initial
>>>>>> version
>>>>>> VERSION = '1.0'
>>>>>>
>>>>>> fields = {
>>>>>> 'resources':
>>>>>>
>>> ListOfObjectsField(ResourceLimit),
>>>>>> }
>>>>>>
>>>>>>
>>> @cache_this_heavily
>>> @remotable_classmethod
>>>>>> def
>>>>>>
>>> get_all_by_user(cls,
>>>>>> user_uuid):
>>>>>> """Returns a
>>> Limits
>>>>>> object that tells
>>>>>> the caller what a
>>>>>> user's
>>>>>> absolute limits
>>> for
>>>>>> the set of
>>> resource
>>>>>> classes in the
>>>>>> system.
>>>>>> """
>>>>>> # Grab a keystone
>>>>>> client session
>>>>>> object and
>>> connect
>>>>>> to Keystone
>>>>>> ks =
>>>>>>
>>> ksclient.Session(...)
>>>>>> raw_limits =
>>>>>>
>>> ksclient.get_limits_by_user()
>>>>>> return
>>>>>>
>>> cls(resources=[ResourceLimit(**d) for d in raw_limits])
>>>>>> 3) Each service
>>>>>> project would be
>>>>>> responsible for
>>>>>> handling the
>>>>>> consumption of a
>>> set
>>>>>> of requested
>>>>>> resource amounts
>>> in
>>>>>> an atomic and
>>>>>>
>>>>>>
>>>>>> consistent way.
>>>>>>
>>>>>>
>>>>>> [amrith] This is where
>>> the
>>>>>> rubber meets the road.
>>> What
>>>>>> is that atomic
>>>>>>
>>>>>>
>>>>>>
>>>>>> and consistent way? And what
>>>>>> computing infrastructure do you
>>> need
>>>>>> to
>>>>>> deliver this?
>>>>>>
>>>>>> The Delimiter
>>>>>> library would
>>> return
>>>>>> the limits that
>>> the
>>>>>> service would
>>>>>> pre- check before
>>>>>> claiming the
>>>>>> resources and
>>> either
>>>>>> post-check after
>>>>>> claim or utilize
>>> a
>>> compare-and-update
>>>>>> technique with a
>>>>>>
>>> generation/timestamp
>>>>>> during claiming
>>> to
>>>>>> prevent race
>>>>>> conditions.
>>>>>>
>>>>>> For instance, in
>>>>>> Nova with the new
>>>>>> resource
>>> providers
>>>>>> database schema
>>>>>> and doing claims
>>> in
>>>>>> the scheduler (a
>>>>>> proposed change),
>>> we
>>>>>> might do
>>>>>> something to the
>>>>>> effect of:
>>>>>>
>>>>>> from delimiter
>>>>>> import objects as
>>>>>> delim_obj from
>>>>>> delimier import
>>>>>> exceptions as
>>>>>> delim_exc from
>>> nova
>>>>>> import objects as
>>>>>> nova_obj
>>>>>>
>>>>>> request =
>>>>>>
>>> nova_obj.RequestSpec.get_by_uuid(request_uuid)
>>>>>> requested =
>>>>>> request.resources
>>>>>> limits =
>>>>>>
>>> delim_obj.ResourceLimitList.get_all_by_user(user_uuid)
>>>>>> allocations =
>>>>>>
>>> nova_obj.AllocationList.get_all_by_user(user_uuid)
>>>>>> # Pre-check for
>>>>>> violations
>>>>>> for
>>> resource_class,
>>>>>> requested_amount
>>> in
>>> requested.items():
>>>>>> limit_idx =
>>>>>>
>>> limits.resources.index(resource_class)
>>>>>> resource_limit =
>>>>>>
>>> limits.resources[limit_idx].amount
>>>>>> alloc_idx =
>>>>>>
>>> allocations.resources.index(resource_class)
>>>>>> resource_used =
>>>>>>
>>> allocations.resources[alloc_idx]
>>>>>> if (resource_used
>>> +
>>> requested_amount)>
>>>>>> resource_limit:
>>>>>> raise
>>>>>>
>>> delim_exc.QuotaExceeded
>>>>>>
>>>>>> [amrith] Is the above
>>> code
>>>>>> run with some global
>>> mutex
>>>>>> to prevent that
>>>>>>
>>>>>>
>>>>>>
>>>>>> two people don't believe that
>>> they
>>>>>> are good on quota at the same
>>> time?
>>>>>>
>>>>>> # Do claims in
>>>>>> scheduler in an
>>>>>> atomic,
>>> consistent
>>>>>> fashion...
>>>>>> claims =
>>>>>>
>>> scheduler_client.claim_resources(request)
>>>>>>
>>>>>> [amrith] Yes, each
>>> 'atomic'
>>>>>> claim on a
>>> repeatable-read
>>>>>> database could
>>>>>>
>>>>>>
>>>>>>
>>>>>> result in oversubscription.
>>>>>>
>>>>>>
>>>>>> # Post-check for
>>>>>> violations
>>>>>> allocations =
>>>>>>
>>> nova_obj.AllocationList.get_all_by_user(user_uuid)
>>>>>> # allocations now
>>>>>> include the
>>> claimed
>>>>>> resources from
>>> the
>>>>>> scheduler
>>>>>>
>>>>>> for
>>> resource_class,
>>>>>> requested_amount
>>> in
>>> requested.items():
>>>>>> limit_idx =
>>>>>>
>>> limits.resources.index(resource_class)
>>>>>> resource_limit =
>>>>>>
>>> limits.resources[limit_idx].amount
>>>>>> alloc_idx =
>>>>>>
>>> allocations.resources.index(resource_class)
>>>>>> resource_used =
>>>>>>
>>> allocations.resources[alloc_idx]
>>>>>> if resource_used>
>>>>>> resource_limit:
>>>>>> # Delete the
>>>>>> allocation
>>> records
>>>>>> for the resources
>>>>>> just claimed
>>>>>>
>>> delete_resources(claims)
>>>>>> raise
>>>>>>
>>> delim_exc.QuotaExceeded
>>>>>>
>>>>>> [amrith] Again, two
>>> people
>>>>>> could drive through this
>>>>>> code and both of
>>>>>> them could fail :(
>>>>>>
>>>>>> 4) The only other
>>>>>> thing that would
>>>>>> need to be done
>>> for
>>>>>> a first go of
>>>>>> the Delimiter
>>>>>> library is some
>>>>>> event listener
>>> that
>>>>>> can listen for
>>>>>> changes to the
>>> quota
>>>>>> limits for a
>>>>>>
>>> user/tenant/default
>>>>>> in Keystone.
>>>>>> We'd want the
>>>>>> services to be
>>> able
>>>>>> notify someone if
>>> a
>>>>>> reduction in
>>>>>> quota results in
>>> an
>>>>>> overquota
>>> situation.
>>>>>> Anyway, that's my
>>>>>> idea. Keep the
>>>>>> Delimiter library
>>>>>> small and focused
>>>>>> on describing the
>>>>>> limits only, not
>>> on
>>>>>> the resource
>>>>>> allocations. Have
>>>>>> the Delimiter
>>>>>> library present a
>>>>>> versioned object
>>>>>> interface so the
>>>>>> interaction
>>> between
>>>>>> the data exposed
>>> by
>>>>>> the Keystone REST
>>>>>> API for
>>>>>> quotas can evolve
>>>>>> naturally and
>>>>>> smoothly over
>>> time.
>>>>>> Best,
>>>>>> -jay
>>>>>>
>>>>>> Let me
>>>>>>
>>> illustrate
>>>>>> the way I
>>>>>> see these
>>>>>> things
>>>>>> fitting
>>>>>> together.
>>> A
>>> hypothetical
>>>>>> Trove
>>> system
>>>>>> may be
>>> setup
>>>>>> as
>>> follows:
>>>>>> - No more
>>>>>> than 2000
>>>>>> database
>>>>>> instances
>>> in
>>>>>> total,
>>> 300
>>>>>> clusters
>>>>>>
>>>>>>
>>>>>> in
>>>>>>
>>>>>>
>>>>>>
>>>>>> total
>>>>>> - Users
>>> may
>>>>>> not
>>> launch
>>>>>> more than
>>> 25
>>>>>> database
>>>>>>
>>> instances,
>>>>>> or 4
>>>>>> clusters
>>>>>> - The
>>>>>>
>>> particular
>>>>>> user
>>>>>> 'amrith'
>>> is
>>>>>> limited
>>> to 2
>>>>>> databases
>>>>>> and
>>>>>>
>>>>>>
>>>>>> 1
>>>>>>
>>>>>>
>>>>>>
>>>>>> cluster
>>>>>> - No user
>>>>>> may
>>> consume
>>>>>> more than
>>>>>> 20TB of
>>>>>> storage
>>> at a
>>>>>> time
>>>>>> - No user
>>>>>> may
>>> consume
>>>>>> more than
>>>>>> 10GB of
>>>>>> memory at
>>> a
>>>>>> time
>>>>>>
>>>>>> At
>>> startup,
>>>>>> I believe
>>>>>> that the
>>>>>> system
>>> would
>>>>>> make the
>>>>>> following
>>>>>> sequence
>>> of
>>>>>> calls:
>>>>>>
>>>>>> -
>>>>>>
>>> define_resource(databaseInstance, 2000, 0, 25, 0, ...)
>>>>>> -
>>>>>>
>>> update_resource_limits(databaseInstance, amrith, 2, 0,
>>>>>>
>>>>>> ...)
>>>>>>
>>>>>>
>>>>>>
>>> -
>>> define_resource(databaseCluster, 300, 0, 4, 0, ...)
>>>>>> -
>>>>>>
>>> update_resource_limits(databaseCluster, amrith, 1, 0, ...)
>>>>>> -
>>>>>>
>>> define_resource(storage, -1, 0, 20TB, 0, ...)
>>>>>> -
>>>>>>
>>> define_resource(memory, -1, 0, 10GB, 0, ...)
>>>>>> Assume
>>> that
>>>>>> the user
>>>>>> john
>>> comes
>>>>>> along and
>>>>>> asks for
>>> a
>>>>>> cluster
>>> with
>>>>>> 4
>>>>>> nodes,
>>> 1TB
>>>>>> storage
>>> per
>>>>>> node and
>>>>>> each node
>>>>>> having
>>> 1GB
>>>>>> of
>>> memory,
>>>>>> the
>>>>>> system
>>> would
>>>>>> go
>>> through
>>>>>> the
>>>>>> following
>>>>>> sequence:
>>>>>>
>>>>>> -
>>>>>>
>>> reserve_resource(databaseCluster, john, 1, None)
>>>>>> o this
>>>>>> returns a
>>>>>>
>>> resourceID
>>>>>> (say
>>>>>>
>>> cluster-resource-
>>>>>>
>>>>>> ID)
>>>>>>
>>>>>>
>>>>>>
>>>>>> o
>>> the
>>>>>> cluster
>>>>>> instance
>>>>>> that it
>>>>>> reserves
>>>>>> counts
>>>>>>
>>>>>>
>>>>>>
>>>>>> against
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> the
>>>>>> limit of
>>> 300
>>>>>> cluster
>>>>>> instances
>>> in
>>>>>> total, as
>>>>>> well
>>>>>>
>>>>>>
>>>>>>
>>>>>> as
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> the 4
>>>>>> clusters
>>>>>> that john
>>>>>> can
>>>>>>
>>> provision.
>>>>>> If
>>> 'amrith'
>>>>>>
>>>>>>
>>>>>> had
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> requested
>>>>>> it, that
>>>>>> would
>>> have
>>>>>> been
>>> counted
>>>>>> against
>>>>>>
>>>>>>
>>>>>>
>>>>>> the
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> limit
>>>>>> of 2
>>>>>> clusters
>>> for
>>>>>> the user.
>>>>>>
>>>>>> -
>>>>>>
>>> reserve_resource(databaseInstance, john, 1,
>>> cluster-resource-id)
>>>>>> -
>>>>>>
>>> reserve_resource(databaseInstance, john, 1,
>>> cluster-resource-id)
>>>>>> -
>>>>>>
>>> reserve_resource(databaseInstance, john, 1,
>>> cluster-resource-id)
>>>>>> -
>>>>>>
>>> reserve_resource(databaseInstance, john, 1,
>>> cluster-resource-id)
>>>>>> o this
>>>>>> returns
>>> four
>>>>>> resource
>>>>>> id's,
>>> let's
>>>>>> say
>>>>>>
>>> instance-1-id, instance-2-id, instance-3-id,
>>> instance-4-id
>>>>>> o note
>>> that
>>>>>> each
>>>>>> instance
>>> is
>>>>>> that, an
>>>>>> instance
>>> by
>>>>>> itself.
>>> it
>>>>>> is
>>> therefore
>>>>>> not right
>>> to
>>>>>> consider
>>>>>> this
>>>>>>
>>>>>>
>>>>>> as
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> equivalent
>>>>>> to a call
>>> to
>>> reserve_resource() with a
>>>>>>
>>>>>>
>>>>>> size
>>>>>>
>>>>>>
>>>>>>
>>>>>> of
>>> 4,
>>> especially
>>>>>> because
>>> each
>>>>>> instance
>>>>>> could
>>> later
>>>>>>
>>>>>>
>>>>>> be
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> tracked
>>> as
>>>>>> an
>>>>>>
>>> individual
>>>>>> Nova
>>>>>> instance.
>>>>>>
>>>>>> -
>>>>>>
>>> reserve_resource(storage, john, 1TB, instance-1-id)
>>>>>> -
>>>>>>
>>> reserve_resource(storage, john, 1TB, instance-2-id)
>>>>>> -
>>>>>>
>>> reserve_resource(storage, john, 1TB, instance-3-id)
>>>>>> -
>>>>>>
>>> reserve_resource(storage, john, 1TB, instance-4-id)
>>>>>> o each of
>>>>>> them
>>> returns
>>>>>> some
>>>>>>
>>> resourceID,
>>>>>> let's say
>>>>>>
>>>>>>
>>>>>> they
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> returned
>>>>>>
>>> cinder-1-id,
>>> cinder-2-id,
>>> cinder-3-id,
>>> cinder-4-id
>>>>>> o since
>>> the
>>>>>> storage
>>> of
>>>>>> 1TB is a
>>>>>> unit, it
>>> is
>>>>>> treated
>>>>>>
>>>>>>
>>>>>> as
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> such.
>>>>>> In other
>>>>>> words,
>>> you
>>>>>> don't
>>> need
>>>>>> to invoke
>>>>>>
>>> reserve_resource 10^12 times, once per byte
>>>>>> allocated
>>>>>> :)
>>>>>>
>>>>>> -
>>>>>>
>>> reserve_resource(memory, john, 1GB, instance-1-id)
>>>>>> -
>>>>>>
>>> reserve_resource(memory, john, 1GB, instance-2-id)
>>>>>> -
>>>>>>
>>> reserve_resource(memory, john, 1GB, instance-3-id)
>>>>>> -
>>>>>>
>>> reserve_resource(memory, john, 1GB, instance-4-id)
>>>>>> o each of
>>>>>> these
>>> return
>>> something,
>>>>>> say
>>>>>>
>>> Dg4KBQcODAENBQEGBAcEDA, CgMJAg8FBQ8GDwgLBA8FAg,
>>> BAQJBwYMDwAIAA0DBAkNAg, AQMLDA4OAgEBCQ0MBAMGCA. I
>>>>>>
>>>>>> have
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> made
>>>>>> up
>>> arbitrary
>>>>>> strings
>>> just
>>>>>> to
>>> highlight
>>>>>> that we
>>>>>> really
>>> don't
>>>>>> track
>>> these
>>>>>> anywhere
>>> so
>>>>>> we don't
>>>>>> care
>>>>>>
>>>>>>
>>>>>> about
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> them.
>>>>>> If all
>>> this
>>>>>> works,
>>> then
>>>>>> the
>>> system
>>>>>> knows
>>> that
>>>>>> John's
>>>>>> request
>>> does
>>>>>> not
>>> violate
>>>>>> any
>>> quotas
>>>>>> that it
>>> can
>>>>>> enforce,
>>> it
>>>>>> can then
>>> go
>>>>>> ahead and
>>>>>> launch
>>> the
>>>>>> instances
>>>>>> (calling
>>>>>> Nova),
>>>>>> provision
>>>>>> storage,
>>> and
>>>>>> so on.
>>>>>>
>>>>>> The
>>> system
>>>>>> then goes
>>>>>> and
>>> creates
>>>>>> four
>>> Cinder
>>>>>> volumes,
>>>>>> these are
>>>>>>
>>> cinder-1-uuid, cinder-2-uuid, cinder-3-uuid, cinder-4-uuid.
>>>>>> It can
>>> then
>>>>>> go and
>>>>>> confirm
>>>>>> those
>>>>>>
>>> reservations.
>>>>>> -
>>>>>>
>>> provision_resource(cinder-1-id, cinder-1-uuid)
>>>>>> -
>>>>>>
>>> provision_resource(cinder-2-id, cinder-2-uuid)
>>>>>> -
>>>>>>
>>> provision_resource(cinder-3-id, cinder-3-uuid)
>>>>>> -
>>>>>>
>>> provision_resource(cinder-4-id, cinder-4-uuid)
>>>>>> It could
>>>>>> then go
>>> and
>>>>>> launch 4
>>>>>> nova
>>>>>> instances
>>>>>> and
>>>>>> similarly
>>>>>> provision
>>>>>> those
>>>>>>
>>> resources,
>>>>>> and so
>>> on.
>>>>>> This
>>> process
>>>>>> could
>>> take
>>>>>> some
>>> minutes
>>>>>> and
>>>>>> holding a
>>>>>> database
>>>>>>
>>> transaction
>>>>>> open for
>>>>>> this is
>>> the
>>>>>> issue
>>> that
>>>>>> Jay
>>>>>> brings up
>>> in
>>>>>> [4]. We
>>>>>> don't
>>> have
>>>>>> to in
>>> this
>>>>>> proposed
>>>>>> scheme.
>>>>>>
>>>>>> Since the
>>>>>> resources
>>>>>> are all
>>>>>>
>>> hierarchically linked through the
>>>>>> overall
>>>>>> cluster
>>> id,
>>>>>> when the
>>>>>> cluster
>>> is
>>>>>> setup, it
>>>>>> can
>>> finally
>>>>>> go and
>>>>>> provision
>>>>>> that:
>>>>>>
>>>>>> -
>>>>>>
>>> provision_resource(cluster-resource-id, cluster-uuid)
>>>>>> When
>>> Trove
>>>>>> is done
>>> with
>>>>>> some
>>>>>>
>>> individual
>>>>>> resource,
>>> it
>>>>>> can go
>>> and
>>>>>> release
>>> it.
>>>>>> Note that
>>>>>> I'm
>>> thinking
>>>>>> this will
>>>>>> invoke
>>>>>>
>>> release_resource
>>>>>> with the
>>> ID
>>>>>> of the
>>>>>>
>>> underlying
>>>>>> object OR
>>>>>> the
>>>>>> resource.
>>>>>>
>>>>>> -
>>>>>>
>>> release_resource(cinder-4-id), and
>>>>>> -
>>>>>>
>>> release_resource(cinder-4-uuid)
>>>>>> are
>>>>>> therefore
>>>>>> identical
>>>>>> and
>>> indicate
>>>>>> that the
>>> 4th
>>>>>> 1TB
>>> volume
>>>>>> is now
>>>>>> released.
>>>>>> How this
>>>>>> will be
>>>>>>
>>> implemented
>>>>>> in
>>> Python,
>>>>>> kwargs or
>>>>>> some
>>>>>> other
>>>>>> mechanism
>>>>>> is, I
>>>>>> believe,
>>> an
>>> implementation detail.
>>>>>> Finally,
>>> it
>>>>>> releases
>>> the
>>>>>> cluster
>>>>>> resource
>>> by
>>>>>> doing
>>> this:
>>>>>> -
>>>>>>
>>> release_resource(cluster-resource-id)
>>>>>> This
>>> would
>>>>>> release
>>> the
>>>>>> cluster
>>> and
>>>>>> all
>>>>>> dependent
>>>>>> resources
>>> in
>>>>>> a
>>>>>> single
>>>>>>
>>> operation.
>>>>>> A user
>>> may
>>>>>> wish to
>>>>>> manage a
>>>>>> resource
>>>>>> that was
>>>>>>
>>> provisioned
>>>>>> from the
>>>>>> service.
>>>>>> Assume
>>> that
>>>>>> this
>>> results
>>>>>> in a
>>>>>> resizing
>>> of
>>>>>> the
>>>>>>
>>> instances,
>>>>>> then it
>>> is a
>>>>>> matter of
>>>>>> updating
>>>>>> that
>>>>>> resource.
>>>>>>
>>>>>> Assume
>>> that
>>>>>> the third
>>>>>> 1TB
>>> volume
>>>>>> is being
>>>>>> resized
>>> to
>>>>>> 2TB, then
>>> it
>>>>>> is
>>>>>> merely a
>>>>>> matter of
>>>>>> invoking:
>>>>>>
>>>>>> -
>>>>>>
>>> update_resource(cinder-3-uuid, 2TB)
>>>>>> Delimiter
>>>>>> can go
>>>>>> figure
>>> out
>>>>>> that
>>>>>>
>>> cinder-3-uuid is a 1TB device and
>>>>>> therefore
>>>>>> this is
>>> an
>>>>>> increase
>>> of
>>>>>> 1TB and
>>>>>> verify
>>> that
>>>>>> this is
>>>>>> within
>>>>>> the
>>> quotas
>>>>>> allowed
>>> for
>>>>>> the user.
>>>>>>
>>>>>> The thing
>>>>>> that I
>>> find
>>> attractive
>>>>>> about
>>> this
>>>>>> model of
>>>>>>
>>> maintaining
>>>>>> a
>>>>>> hierarchy
>>> of
>>> reservations
>>>>>> is that
>>> in
>>>>>> the event
>>> of
>>>>>> an error,
>>>>>> the
>>>>>> service
>>> need
>>>>>> merely
>>> call
>>> release_resource() on the highest level
>>> reservation
>>>>>> and the
>>>>>> Delimiter
>>>>>> project
>>> can
>>>>>> walk down
>>>>>> the chain
>>>>>> and
>>>>>> release
>>> all
>>>>>> the
>>>>>> resources
>>> or
>>> reservations
>>>>>> as
>>>>>>
>>> appropriate.
>>>>>> Under the
>>>>>> covers I
>>>>>> believe
>>> that
>>>>>> each of
>>>>>> these
>>>>>>
>>> operations
>>>>>> should be
>>>>>> atomic
>>> and
>>>>>> may
>>> update
>>>>>> multiple
>>>>>> database
>>>>>> tables
>>> but
>>>>>> these
>>> will
>>>>>> all be
>>>>>> short
>>> lived
>>> operations.
>>>>>> For
>>> example,
>>>>>> reserving
>>> an
>>>>>> instance
>>>>>> resource
>>>>>> would
>>>>>> increment
>>>>>> the
>>>>>> number of
>>>>>> instances
>>>>>> for the
>>> user
>>>>>> as well
>>> as
>>>>>> the
>>> number
>>>>>> of
>>> instances
>>>>>> on the
>>>>>> whole,
>>> and
>>>>>> this
>>> would
>>>>>> be an
>>> atomic
>>> operation.
>>>>>> I have
>>> two
>>>>>> primary
>>>>>> areas of
>>>>>> concern
>>>>>> about the
>>>>>> proposal
>>>>>> [3].
>>>>>>
>>>>>> The first
>>> is
>>>>>> that it
>>>>>> makes the
>>>>>> implicit
>>>>>>
>>> assumption
>>>>>> that the
>>>>>> "flat
>>> mode"
>>>>>> is
>>>>>>
>>> implemented.
>>>>>> That
>>>>>> provides
>>>>>> value to
>>> a
>>>>>>
>>>>>> consumer
>>>>>>
>>>>>>
>>>>>>
>>>>>> but I
>>> think
>>>>>> it leaves
>>> a
>>>>>> lot for
>>> the
>>>>>> consumer
>>> to
>>>>>> do. For
>>>>>>
>>>>>>
>>>>>>
>>>>>> example,
>>>>>>
>>>>>>
>>>>>>
>>> I
>>>>>> find it
>>> hard
>>>>>> to see
>>> how
>>>>>> the model
>>>>>> proposed
>>>>>> would
>>> handle
>>>>>>
>>>>>>
>>>>>> the
>>>>>>
>>>>>>
>>>>>>
>>>>>> release
>>> of
>>>>>> quotas,
>>>>>> leave
>>> alone
>>>>>> the case
>>> of
>>>>>> a nested
>>>>>> release
>>> of
>>>>>>
>>>>>> a
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> hierarchy
>>>>>> of
>>>>>>
>>> resources.
>>>>>> The other
>>> is
>>>>>> the
>>> notion
>>>>>> that the
>>>>>>
>>> implementation will begin a
>>> transaction,
>>>>>> perform a
>>>>>> query(),
>>>>>> make some
>>>>>>
>>> manipulations, and
>>>>>> then do a
>>>>>> save().
>>> This
>>>>>> makes for
>>> an
>>> interesting
>>> transaction
>>> management
>>>>>> challenge
>>> as
>>>>>> it would
>>>>>> require
>>> the
>>> underlying
>>>>>>
>>>>>> database
>>>>>>
>>>>>>
>>>>>>
>>>>>> to run
>>> in
>>>>>> an
>>> isolation
>>>>>> mode of
>>> at
>>>>>> least
>>>>>>
>>> repeatable
>>>>>> reads and
>>>>>> maybe
>>> even
>>> serializable
>>>>>> which
>>> would
>>>>>> be a
>>>>>>
>>> performance
>>>>>> bear on
>>>>>>
>>>>>>
>>>>>> a
>>>>>>
>>>>>>
>>>>>>
>>>>>> heavily
>>>>>> loaded
>>>>>> system.
>>> If
>>>>>> run in
>>> the
>>> traditional
>>>>>> read-
>>>>>>
>>>>>>
>>>>>>
>>>>>> committed
>>>>>>
>>>>>>
>>>>>>
>>>>>> mode,
>>> this
>>>>>> would
>>>>>> silently
>>>>>> lead to
>>> over
>>> subscriptions, and
>>>>>>
>>>>>>
>>>>>> the
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> violation
>>>>>> of quota
>>>>>> limits.
>>>>>>
>>>>>> I believe
>>>>>> that it
>>>>>> should be
>>> a
>>> requirement
>>>>>> that the
>>>>>> Delimiter
>>>>>> library
>>>>>> should be
>>>>>> able to
>>> run
>>>>>> against a
>>>>>> database
>>>>>> that
>>>>>> supports,
>>>>>> and is
>>>>>>
>>> configured
>>>>>> for
>>>>>>
>>> READ-COMMITTED, and should not require anything higher.
>>>>>> The model
>>>>>> proposed
>>>>>> above can
>>>>>> certainly
>>> be
>>> implemented
>>>>>> with a
>>>>>> database
>>>>>> running
>>>>>>
>>> READ-COMMITTED, and I believe that this is also
>>>>>> true with
>>>>>> the
>>> caveat
>>>>>> that the
>>>>>>
>>> operations
>>>>>> will be
>>>>>> performed
>>>>>> through
>>>>>>
>>>>>>
>>>>>> SQLAlchemy.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> -amrith
>>>>>>
>>>>>> [1]
>>>>>>
>>> http://openstack.markmail.org/thread/tkl2jcyvzgifniux
>>>>>> [2]
>>>>>>
>>> http://openstack.markmail.org/thread/3cr7hoeqjmgyle2j
>>>>>> [3]
>>>>>>
>>> https://review.openstack.org/#/c/284454/
>>>>>> [4]
>>>>>>
>>> http://markmail.org/message/7ixvezcsj3uyiro6
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> ____________________________________________________________________
>>>>>> __ ____
>>>>>> OpenStack
>>>>>>
>>> Development
>>>>>> Mailing
>>> List
>>>>>> (not for
>>>>>> usage
>>>>>>
>>> questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>> _____________________________________________________________________
>>>>>> _____ OpenStack
>>>>>> Development
>>> Mailing
>>>>>> List (not for
>>> usage
>>>>>> questions)
>>>>>> Unsubscribe:
>>>>>>
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>> ______________________________________________________________________
>>>>>> ____ OpenStack
>>> Development
>>>>>> Mailing List (not for
>>> usage
>>>>>> questions)
>>>>>> Unsubscribe:
>>>>>>
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>> __________________________________________________________________________
>>>
>>>>>> OpenStack Development Mailing
>>> List
>>>>>> (not for usage questions)
>>>>>> Unsubscribe:
>>>>>>
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>>>>>
>>> __________________________________________________________________________
>>>
>>>>>> OpenStack Development Mailing List (not
>>> for
>>>>>> usage questions)
>>>>>> Unsubscribe:
>>>>>>
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>>>>>
>>> __________________________________________________________________________
>>>
>>>>>> OpenStack Development Mailing List (not for usage
>>>>>> questions)
>>>>>> Unsubscribe:
>>>>>>
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> __________________________________________________________________________
>>>
>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>
>>>>>> Unsubscribe:
>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>
>>> __________________________________________________________________________
>>>
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> __________________________________________________________________________
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
More information about the OpenStack-dev
mailing list