[openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal
Amrith Kumar
amrith at tesora.com
Tue Apr 26 13:08:27 UTC 2016
I would strongly vote against the generation being exposed outside of the library.
-amrith
> -----Original Message-----
> From: Joshua Harlow [mailto:harlowja at fastmail.com]
> Sent: Monday, April 25, 2016 3:06 PM
> To: OpenStack Development Mailing List (not for usage questions)
> <openstack-dev at lists.openstack.org>
> Cc: vilobhmm at yahoo-inc.com; ed at leafe.com
> Subject: Re: [openstack-dev] More on the topic of DELIMITER, the Quota
> Management Library proposal
>
> Is the generation stuff going to be exposed outside of the AP?
>
> I'm sort of hoping not(?), because a service (if one everyone wanted to
> create it) for say zookeeper (or etcd or consul...) could use its
> built-in generation equivalent (every znode has a version that u can use
> to do equivalent things).
>
> Thus it's like the gist I posted earlier @
>
> https://gist.github.com/harlowja/e7175c2d76e020a82ae94467a1441d85
>
> So might be nice to not expose such a thing outside of the db-layer.
>
> Amrith Kumar wrote:
> > On Sat, 2016-04-23 at 21:41 +0000, Amrith Kumar wrote:
> >> Ok to beer and high bandwidth. FYI Jay the distributed high perf db we
> >> did a couple of years ago is now open source. Just saying. Mysql plug
> >> compatible ....
> >
> >> -amrith
> >>
> >>
> >> --
> >> Amrith Kumar
> >> amrith at tesora.com
> >>
> >>
> >> -------- Original message --------
> >> From: Jay Pipes<jaypipes at gmail.com>
> >> Date: 04/23/2016 4:10 PM (GMT-05:00)
> >> To: Amrith Kumar<amrith at tesora.com>,
> >> openstack-dev at lists.openstack.org
> >> Cc: vilobhmm at yahoo-inc.com, nik.komawar at gmail.com, Ed Leafe
> >> <ed at leafe.com>
> >> Subject: Re: [openstack-dev] More on the topic of DELIMITER, the Quota
> >> Management Library proposal
> >>
> >>
> >> Looking forward to arriving in Austin so that I can buy you a beer,
> >> Amrith, and have a high-bandwidth conversation about how you're
> >> wrong. :P
> >
> >
> > Jay and I chatted and it took a long time to come to an agreement
> > because we weren't able to find any beer.
> >
> > Here's what I think we've agreed about. The library will store data in
> > two tables;
> >
> > 1. the detail table which stores the individual claims and the resource
> > class.
> > 2. a generation table which stores the resource class and a generation.
> >
> > When a claim is received, the requestor performs the following
> > operations.
> >
> > begin
> >
> > select sum(detail.claims) as total_claims,
> > generation.resource as resource,
> > generation.generation last_generation
> > from detail, generation
> > where detail.resource = generation.resource
> > and generation.resource =<chosen resource; memory, cpu, ...>
> > group by generation.generation, generation.resource
> >
> > if total_claims + this_claim< limit
> > insert into detail values (this_claim, resource)
> >
> > update generation
> > set generation = generation + 1
> > where generation = last_generation
> >
> > if @@rowcount = 1
> > -- all good
> > commit
> > else
> > rollback
> > -- try again
> >
> >
> >
> > There will be some bootstrapping that will be required for the situation
> > where there are no detail records for a given resource and so on but I
> > think we can figure that out easily. The easiest way I can think of
> > doing that is to lose the join and do both the queries (one against the
> > detail and one against the generation table within the same
> > transaction).
> >
> > Using the generation table update as the locking mechanism that prevents
> > multiple requestors from making concurrent claims.
> >
> > So long as people don't go and try and read these tables and change
> > tables outside of the methods that the library provides, we can
> > guarantee that this is al safe and will not oversubscribe.
> >
> > -amrith
> >
> >> Comments inline.
> >>
> >> On 04/23/2016 11:25 AM, Amrith Kumar wrote:
> >>> On Sat, 2016-04-23 at 10:26 -0400, Andrew Laski wrote:
> >>>> On Fri, Apr 22, 2016, at 09:57 PM, Tim Bell wrote:
> >>>>
> >>>>> I have reservations on f and g.
> >>>>>
> >>>>>
> >>>>> On f., We have had a number of discussions in the past about
> >>>>> centralising quota (e.g. Boson) and the project teams of the other
> >>>>> components wanted to keep the quota contents ‘close’. This can
> >>>>> always be reviewed further with them but I would hope for at least
> >> a
> >>>>> standard schema structure of tables in each project for the
> >> handling
> >>>>> of quota.
> >>>>>
> >>>>>
> >>>>> On g., aren’t all projects now nested projects ? If we have the
> >>>>> complexity of handling nested projects sorted out in the common
> >>>>> library, is there a reason why a project would not want to support
> >>>>> nested projects ?
> >>>>>
> >>>>>
> >>>>> One other issue is how to do reconcilliation, each project needs
> >> to
> >>>>> have a mechanism to re-calculate the current allocations and
> >>>>> reconcile that with the quota usage. While in an ideal world, this
> >>>>> should not be necessary, it would be for the foreseeable future,
> >>>>> especially with a new implementation.
> >>>>>
> >>>> One of the big reasons that Jay and I have been pushing to remove
> >>>> reservations and tracking of quota in a separate place than the
> >>>> resources are actually used, e.g., an instance record in the Nova
> >> db,
> >>>> is so that reconciliation is not necessary. For example, if RAM
> >> quota
> >>>> usage is simply tracked as sum(instances.memory_mb) then you can be
> >>>> sure that usage is always up to date.
> >>> Uh oh, there be gremlins here ...
> >>>
> >>> I am positive that this will NOT work, see earlier conversations
> >> about
> >>> isolation levels, and Jay's alternate solution.
> >>>
> >>> The way (I understand the issue, and Jay's solution) you get around
> >> the
> >>> isolation levels trap is to NOT do your quota determinations based
> >> on a
> >>> SUM(column) but rather based on the rowcount on a well crafted
> >> UPDATE of
> >>> a single table that stored total quota.
> >> No, we would do our quota calculations by doing a SUM(used) against
> >> the
> >> allocations table. There is no separate table that stored the total
> >> quota (or quota usage records). That's the source of the problem with
> >> the existing quota handling code in Nova. The generation field value
> >> is
> >> used to provide the consistent view of the actual resource usage
> >> records
> >> so that the INSERT operations for all claimed resources can be done in
> >> a
> >> transactional manner and will be rolled back if any other writer
> >> changes
> >> the amount of consumed resources on a provider (which of course would
> >> affect the quota check calculations).
> >>
> >> > You could also store a detail
> >>> claim record for each claim in an independent table that is
> >> maintained
> >>> in the same database transaction if you so desire, that is optional.
> >> The allocations table is the "detail claim record" table that you
> >> refer
> >> to above.
> >>
> >>> My view of how this would work (which I described earlier as
> >> building on
> >>> Jay's solution) is that the claim flow would look like this:
> >>>
> >>> select total_used, generation
> >>> from quota_claimed
> >>> where tenant =<tenant> and resource = 'memory'
> >> There is no need to keep a total_used value for anything. That is
> >> denormalized calculated data that merely adds a point of race
> >> contention. The quota check is against the *detail* table
> >> (allocations),
> >> which stores the *actual resource usage records*.
> >>
> >>> begin transaction
> >>>
> >>> update quota_claimed
> >>> set total_used = total_used + claim, generation =
> >>> generation + 1
> >>> where tenant =<tenant> and resource = 'memory'
> >>> and generation = generation
> >>> and total_used + claim< limit
> >> This part of the transaction must always occur **after** the
> >> insertion
> >> of the actual resource records, not before.
> >>
> >>> if @@rowcount = 1
> >>> -- optional claim_detail table
> >>> insert into claim_detail values (<tenant>,
> >> 'memory',
> >>> claim, ...)
> >>> commit
> >>> else
> >>> rollback
> >> So, in pseudo-Python-SQLish code, my solution works like this:
> >>
> >> limits = get_limits_from_delimiter()
> >> requested = get_requested_from_request_spec()
> >>
> >> while True:
> >>
> >> used := SELECT
> >> resource_class,
> >> resource_provider,
> >> generation,
> >> SUM(used) as total_used
> >> FROM allocations
> >> JOIN resource_providers ON (...)
> >> WHERE consumer_uuid = $USER_UUID
> >> GROUP BY
> >> resource_class,
> >> resource_provider,
> >> generation;
> >>
> >> # Check that our requested resource amounts don't exceed quotas
> >> if not check_requested_within_limits(requested, used, limits):
> >> raise QuotaExceeded
> >>
> >> # Claim all requested resources. Note that the generation
> >> retrieved
> >> # from the above query is our consistent view marker. If the
> >> UPDATE
> >> # below succeeds and returns != 0 rows affected, that means there
> >> # was no other writer that changed our resource usage in between
> >> # this thread's claiming of resources, and therefore we prevent
> >> # any oversubscription of resources.
> >> begin_transaction:
> >>
> >> provider := SELECT id, generation, ... FROM
> >> resource_providers
> >> JOIN (...)
> >> WHERE (<resource_usage_filters>)
> >>
> >> for resource in requested:
> >> INSERT INTO allocations (
> >> resource_provider_id,
> >> resource_class_id,
> >> consumer_uuid,
> >> used
> >> ) VALUES (
> >> $provider.id,
> >> $resource.id,
> >> $USER_UUID,
> >> $resource.amount
> >> );
> >>
> >> rows_affected := UPDATE resource_providers
> >> SET generation = generation + 1
> >> WHERE id = $provider.id
> >> AND generation =
> >> $used[$provider.id].generation;
> >>
> >> if $rows_affected == 0:
> >> ROLLBACK;
> >>
> >> The only reason we would need a post-claim quota check is if some of
> >> the
> >> requested resources are owned and tracked by an external-to-Nova
> >> system.
> >>
> >> BTW, note to Ed Leafe... unless your distributed data store supports
> >> transactional semantics, you can't use a distributed data store for
> >> these types of solutions. Instead, you will need to write a whole
> >> bunch
> >> of code that does post-auditing of claims and quotas and a system
> >> that
> >> accepts that oversubscription and out-of-sync quota limits and usages
> >> is
> >> a fact of life. Not to mention needing to implement JOINs in Python.
> >>
> >>> But, it is my understanding that
> >>>
> >>> (a) if you wish to do the SUM(column) approach that you
> >> propose,
> >>> you must have a reservation that is committed and then you
> >> must
> >>> re-read the SUM(column) to make sure you did not
> >> over-subscribe;
> >>> and
> >> Erm, kind of? Oversubscription is not possible in the solution I
> >> describe because the compare-and-update on the
> >> resource_providers.generation field allows for a consistent view of
> >> the
> >> resources used -- and if that view changes during the insertion of
> >> resource usage records -- the transaction containing those insertions
> >> is
> >> rolled back.
> >>
> >>> (b) to get away from reservations you must stop using the
> >>> SUM(column) approach and instead use a single quota_claimed
> >>> table to determine the current quota claimed.
> >> No. This has nothing to do with reservations.
> >>
> >>> At least that's what I understand of Jay's example from earlier in
> >> this
> >>> thread.
> >>>
> >>> Let's definitely discuss this in Austin. While I don't love Jay's
> >>> solution for other reasons to do with making the quota table a
> >> hotspot
> >>> and things like that, it is a perfectly workable solution, I think.
> >> There is no quota table in my solution.
> >>
> >> If you refer to the resource_providers table (the table that has the
> >> generation field), then yes, it's a hot spot. But hot spots in the DB
> >> aren't necessarily a bad thing if you design the underlying schema
> >> properly.
> >>
> >> More in Austin.
> >>
> >> Best,
> >> -jay
> >>
> >>>>
> >>>>> Tim
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> From: Amrith Kumar<amrith at tesora.com>
> >>>>> Reply-To: "OpenStack Development Mailing List (not for usage
> >>>>> questions)"<openstack-dev at lists.openstack.org>
> >>>>> Date: Friday 22 April 2016 at 06:51
> >>>>> To: "OpenStack Development Mailing List (not for usage questions)"
> >>>>> <openstack-dev at lists.openstack.org>
> >>>>> Subject: Re: [openstack-dev] More on the topic of DELIMITER, the
> >>>>> Quota Management Library proposal
> >>>>>
> >>>>>
> >>>>>
> >>>>> I’ve thought more about Jay’s approach to enforcing
> >> quotas
> >>>>> and I think we can build on and around it. With that
> >>>>> implementation as the basic quota primitive, I think we
> >> can
> >>>>> build a quota management API that isn’t dependent on
> >>>>> reservations. It does place some burdens on the consuming
> >>>>> projects that I had hoped to avoid and these will cause
> >>>>> heartburn for some (make sure that you always request
> >>>>> resources in a consistent order and free them in a
> >>>>> consistent order being the most obvious).
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> If it doesn’t make it harder, I would like to see if we
> >> can
> >>>>> make the quota API take care of the ordering of requests.
> >>>>> i.e. if the quota API is an extension of Jay’s example
> >> and
> >>>>> accepts some data structure (dict?) with all the claims
> >> that
> >>>>> a project wants to make for some operation, and then
> >>>>> proceeds to make those claims for the project in the
> >>>>> consistent order, I think it would be of some value.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Beyond that, I’m on board with a-g below,
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> -amrith
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> From: Vilobh Meshram
> >>>>> [mailto:vilobhmeshram.openstack at gmail.com]
> >>>>> Sent: Friday, April 22, 2016 4:08 AM
> >>>>> To: OpenStack Development Mailing List (not for usage
> >>>>> questions)<openstack-dev at lists.openstack.org>
> >>>>> Subject: Re: [openstack-dev] More on the topic of
> >> DELIMITER,
> >>>>> the Quota Management Library proposal
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> I strongly agree with Jay on the points related to "no
> >>>>> reservation" , keeping the interface simple and the role
> >> for
> >>>>> Delimiter (impose limits on resource consumption and
> >> enforce
> >>>>> quotas).
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> The point to keep user quota, tenant quotas in Keystone
> >>>>> sounds interestring and would need support from Keystone
> >>>>> team. We have a Cross project session planned [1] and
> >> will
> >>>>> definitely bring that up in that session.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> The main thought with which Delimiter was formed was to
> >>>>> enforce resource quota in transaction safe manner and do
> >> it
> >>>>> in a cross-project conducive manner and it still holds
> >>>>> true. Delimiters mission is to impose limits on
> >>>>> resource consumption and enforce quotas in transaction
> >> safe
> >>>>> manner. Few key aspects of Delimiter are :-
> >>>>>
> >>>>>
> >>>>>
> >>>>> a. Delimiter will be a new Library and not a Service.
> >>>>> Details covered in spec.
> >>>>>
> >>>>>
> >>>>> b. Delimiter's role will be to impose limits on resource
> >>>>> consumption.
> >>>>>
> >>>>>
> >>>>> c. Delimiter will not be responsible for rate limiting.
> >>>>>
> >>>>>
> >>>>> d. Delimiter will not maintain data for the resources.
> >>>>> Respective projects will take care of keeping,
> >> maintaining
> >>>>> data for the resources and resource consumption.
> >>>>>
> >>>>>
> >>>>> e. Delimiter will not have the concept of "reservations".
> >>>>> Delimiter will read or update the "actual" resource
> >> tables
> >>>>> and will not rely on the "cached" tables. At present, the
> >>>>> quota infrastructure in Nova, Cinder and other projects
> >> have
> >>>>> tables such as reservations, quota_usage, etc which are
> >> used
> >>>>> as "cached tables" to track re
> >>>>>
> >>>>>
> >>>>> f. Delimiter will fetch the information for project
> >> quota,
> >>>>> user quota from a centralized place, say Keystone, or if
> >>>>> that doesn't materialize will fetch default quota values
> >>>>> from respective service. This information will be cached
> >>>>> since it gets updated rarely but read many times.
> >>>>>
> >>>>>
> >>>>> g. Delimiter will take into consideration whether the
> >>>>> project is a Flat or Nested and will make the
> >> calculations
> >>>>> of allocated, available resources. Nested means project
> >>>>> namespace is hierarchical and Flat means project
> >> namespace
> >>>>> is not hierarchical.
> >>>>>
> >>>>>
> >>>>> -Vilobh
> >>>>>
> >>>>>
> >>>>> [1]
> >> https://www.openstack.org/summit/austin-2016/summit-
> schedule/events/9492
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Apr 21, 2016 at 11:08 PM, Joshua Harlow
> >>>>> <harlowja at fastmail.com> wrote:
> >>>>>
> >>>>>
> >>>>> Since people will be on a plane soon,
> >>>>>
> >>>>> I threw this together as a example of a quota
> >> engine
> >>>>> (the zookeeper code does even work, and yes it
> >>>>> provides transactional semantics due to the nice
> >>>>> abilities of zookeeper znode versions[1] and its
> >>>>> inherent consistency model, yippe).
> >>>>>
> >>>>>
> >> https://gist.github.com/harlowja/e7175c2d76e020a82ae94467a1441d85
> >>>>> Someone else can fill in the db quota engine with
> >> a
> >>>>> similar/equivalent api if they so dare, ha. Or
> >> even
> >>>>> feel to say the gist/api above is crap, cause
> >> that's
> >>>>> ok to, lol.
> >>>>>
> >>>>> [1]
> >>>>>
> >> https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#Data
> >> +Access
> >>>>>
> >>>>>
> >>>>> Amrith Kumar wrote:
> >>>>>
> >>>>> Inline below ... thread is too long, will
> >>>>> catch you in Austin.
> >>>>>
> >>>>>
> >>>>> -----Original Message-----
> >>>>> From: Jay Pipes
> >>>>> [mailto:jaypipes at gmail.com]
> >>>>> Sent: Thursday, April 21, 2016
> >> 8:08
> >>>>> PM
> >>>>> To:
> >>>>> openstack-dev at lists.openstack.org
> >>>>> Subject: Re: [openstack-dev] More
> >> on
> >>>>> the topic of DELIMITER, the Quota
> >>>>> Management Library proposal
> >>>>>
> >>>>> Hmm, where do I start... I think
> >> I
> >>>>> will just cut to the two primary
> >>>>> disagreements I have. And I will
> >>>>> top-post because this email is
> >> way
> >>>>> too
> >>>>> big.
> >>>>>
> >>>>> 1) On serializable isolation
> >> level.
> >>>>> No, you don't need it at all to
> >>>>> prevent races in claiming. Just
> >> use
> >>>>> a
> >>>>> compare-and-update with retries
> >>>>> strategy. Proof is here:
> >>>>>
> >>>>>
> >> https://github.com/jaypipes/placement-
> bench/blob/master/placement.py#L97-
> >>>>> L142
> >>>>>
> >>>>> Works great and prevents multiple
> >>>>> writers from oversubscribing any
> >>>>> resource without relying on any
> >>>>> particular isolation level at
> >> all.
> >>>>> The `generation` field in the
> >>>>> inventories table is what allows
> >>>>> multiple
> >>>>> writers to ensure a consistent
> >> view
> >>>>> of the data without needing to
> >> rely
> >>>>> on
> >>>>> heavy lock-based semantics and/or
> >>>>> RDBMS-specific isolation levels.
> >>>>>
> >>>>>
> >>>>>
> >>>>> [amrith] this works for what it is doing,
> >> we
> >>>>> can definitely do this. This will work at
> >>>>> any isolation level, yes. I didn't want
> >> to
> >>>>> go this route because it is going to
> >> still
> >>>>> require an insert into another table
> >>>>> recording what the actual 'thing' is that
> >> is
> >>>>> claiming the resource and that insert is
> >>>>> going to be in a different transaction
> >> and
> >>>>> managing those two transactions was what
> >> I
> >>>>> wanted to avoid. I was hoping to avoid
> >>>>> having two tables tracking claims, one
> >>>>> showing the currently claimed quota and
> >>>>> another holding the things that claimed
> >> that
> >>>>> quota. Have to think again whether that
> >> is
> >>>>> possible.
> >>>>>
> >>>>> 2) On reservations.
> >>>>>
> >>>>> The reason I don't believe
> >>>>> reservations are necessary to be
> >> in
> >>>>> a quota
> >>>>> library is because reservations
> >> add
> >>>>> a concept of a time to a claim of
> >>>>> some
> >>>>> resource. You reserve some
> >> resource
> >>>>> to be claimed at some point in
> >> the
> >>>>> future and release those
> >> resources
> >>>>> at a point further in time.
> >>>>>
> >>>>> Quota checking doesn't look at
> >> what
> >>>>> the state of some system will be
> >> at
> >>>>> some point in the future. It
> >> simply
> >>>>> returns whether the system *right
> >>>>> now* can handle a request *right
> >>>>> now* to claim a set of resources.
> >>>>>
> >>>>> If you want reservation semantics
> >>>>> for some resource, that's totally
> >>>>> cool,
> >>>>> but IMHO, a reservation service
> >>>>> should live outside of the
> >> service
> >>>>> that is
> >>>>> actually responsible for
> >> providing
> >>>>> resources to a consumer.
> >>>>> Merging right-now quota checks
> >> and
> >>>>> future-based reservations into
> >> the
> >>>>> same
> >>>>> library just complicates things
> >>>>> unnecessarily IMHO.
> >>>>>
> >>>>>
> >>>>>
> >>>>> [amrith] extension of the above ...
> >>>>>
> >>>>> 3) On resizes.
> >>>>>
> >>>>> Look, I recognize some users see
> >>>>> some value in resizing their
> >>>>> resources.
> >>>>> That's fine. I personally think
> >>>>> expand operations are fine, and
> >> that
> >>>>> shrink operations are really the
> >>>>> operations that should be
> >> prohibited
> >>>>> in
> >>>>> the API. But, whatever, I'm fine
> >>>>> with resizing of requested
> >> resource
> >>>>> amounts. My big point is if you
> >>>>> don't have a separate table that
> >>>>> stores
> >>>>> quota_usages and instead only
> >> have a
> >>>>> single table that stores the
> >> actual
> >>>>> resource usage records, you don't
> >>>>> have to do *any* quota check
> >>>>> operations
> >>>>> at all upon deletion of a
> >> resource.
> >>>>> For modifying resource amounts
> >> (i.e.
> >>>>> a
> >>>>> resize) you merely need to change
> >>>>> the calculation of requested
> >>>>> resource
> >>>>> amounts to account for the
> >>>>> already-consumed usage amount.
> >>>>>
> >>>>> Bottom line for me: I really
> >> won't
> >>>>> support any proposal for a
> >> complex
> >>>>> library that takes the resource
> >>>>> claim process out of the hands of
> >>>>> the
> >>>>> services that own those
> >> resources.
> >>>>> The simpler the interface of this
> >>>>> library, the better.
> >>>>>
> >>>>>
> >>>>>
> >>>>> [amrith] my proposal would not but this
> >>>>> email thread has got too long. Yes,
> >> simpler
> >>>>> interface, will catch you in Austin.
> >>>>>
> >>>>> Best,
> >>>>> -jay
> >>>>>
> >>>>> On 04/19/2016 09:59 PM, Amrith
> >> Kumar
> >>>>> wrote:
> >>>>>
> >>>>> -----Original
> >>>>> Message-----
> >>>>> From: Jay Pipes
> >>>>>
> >> [mailto:jaypipes at gmail.com]
> >>>>> Sent: Monday,
> >> April
> >>>>> 18, 2016 2:54 PM
> >>>>> To:
> >>>>>
> >> openstack-dev at lists.openstack.org
> >>>>> Subject: Re:
> >>>>> [openstack-dev]
> >> More
> >>>>> on the topic of
> >>>>> DELIMITER, the
> >>>>> Quota Management
> >>>>> Library proposal
> >>>>>
> >>>>> On 04/16/2016
> >> 05:51
> >>>>> PM, Amrith Kumar
> >>>>> wrote:
> >>>>>
> >>>>> If we
> >>>>> therefore
> >>>>> assume
> >> that
> >>>>> this will
> >> be
> >>>>> a Quota
> >>>>>
> >> Management
> >>>>> Library,
> >>>>> it is
> >> safe
> >>>>> to assume
> >>>>> that
> >> quotas
> >>>>> are going
> >> to
> >>>>> be
> >> managed
> >>>>> on a
> >>>>>
> >> per-project
> >>>>> basis,
> >> where
> >> participating projects will use this library.
> >>>>> I believe
> >>>>> that it
> >>>>> stands to
> >>>>> reason
> >> that
> >>>>> any data
> >>>>>
> >> persistence
> >>>>> will
> >>>>> have to
> >> be
> >>>>> in a
> >>>>> location
> >>>>> decided
> >> by
> >>>>> the
> >>>>>
> >> individual
> >>>>> project.
> >>>>>
> >>>>>
> >>>>> Depends on what
> >> you
> >>>>> mean by "any data
> >>>>> persistence". If
> >> you
> >>>>> are
> >>>>> referring to the
> >>>>> storage of quota
> >>>>> values (per user,
> >>>>> per tenant,
> >>>>> global, etc) I
> >> think
> >>>>> that should be
> >> done
> >>>>> by the Keystone
> >>>>> service.
> >>>>> This data is
> >>>>> essentially an
> >>>>> attribute of the
> >>>>> user or the
> >> tenant
> >>>>> or the
> >>>>>
> >>>>>
> >>>>> service endpoint itself (i.e.
> >>>>>
> >>>>>
> >>>>> global defaults).
> >>>>> This data also
> >>>>> rarely changes
> >> and
> >>>>> logically belongs
> >>>>> to the service
> >> that
> >>>>> manages users,
> >>>>> tenants, and
> >> service
> >>>>> endpoints:
> >>>>>
> >>>>>
> >>>>> Keystone.
> >>>>>
> >>>>>
> >>>>> If you are
> >> referring
> >>>>> to the storage of
> >>>>> resource usage
> >>>>> records, yes,
> >>>>> each service
> >> project
> >>>>> should own that
> >> data
> >>>>> (and frankly, I
> >>>>> don't see a
> >>>>> need to persist
> >> any
> >>>>> quota usage data
> >> at
> >>>>> all, as I
> >> mentioned
> >>>>> in a
> >>>>> previous reply to
> >>>>> Attila).
> >>>>>
> >>>>>
> >>>>> [amrith] You make a
> >>>>> distinction that I had
> >> made
> >>>>> implicitly, and it is
> >>>>> important to highlight
> >> it.
> >>>>> Thanks for pointing it
> >> out.
> >>>>> Yes, I meant
> >>>>> both of the above, and as
> >>>>> stipulated. Global
> >> defaults
> >>>>> in keystone
> >>>>> (somehow, TBD) and usage
> >>>>> records, on a per-service
> >>>>> basis.
> >>>>>
> >>>>> That may
> >> not
> >>>>> be a very
> >>>>>
> >> interesting
> >>>>> statement
> >>>>> but the
> >>>>> corollary
> >>>>> is, I
> >>>>> think, a
> >>>>> very
> >>>>>
> >> significant
> >> statement;
> >>>>> it cannot
> >> be
> >>>>> assumed
> >> that
> >>>>> the
> >>>>> quota
> >>>>>
> >> management
> >> information
> >>>>> for all
> >>>>>
> >> participating projects is in
> >>>>> the same
> >>>>> database.
> >>>>>
> >>>>>
> >>>>> It cannot be
> >> assumed
> >>>>> that this
> >>>>> information is
> >> even
> >>>>> in a database at
> >>>>>
> >>>>>
> >>>>>
> >>>>> all...
> >>>>>
> >>>>>
> >>>>> [amrith] I don't follow.
> >> If
> >>>>> the service in question
> >> is
> >>>>> to be scalable,
> >>>>> I think it stands to
> >> reason
> >>>>> that there must be some
> >>>>> mechanism by which
> >>>>> instances of the service
> >> can
> >>>>> share usage records (as
> >> you
> >>>>> refer to
> >>>>> them, and I like that
> >> term).
> >>>>> I think it stands to
> >> reason
> >>>>> that there
> >>>>> must be some database,
> >> no?
> >>>>> A
> >>>>>
> >> hypothetical
> >>>>> service
> >>>>> consuming
> >>>>> the
> >>>>> Delimiter
> >>>>> library
> >>>>> provides
> >>>>>
> >> requesters
> >>>>> with some
> >>>>> widgets,
> >> and
> >>>>> wishes to
> >>>>> track the
> >>>>> widgets
> >> that
> >>>>> it has
> >>>>>
> >> provisioned
> >>>>> both on a
> >>>>> per-user
> >>>>> basis,
> >> and
> >>>>> on the
> >>>>> whole. It
> >>>>> should
> >>>>> therefore
> >>>>>
> >> multi-tenant
> >>>>> and able
> >> to
> >>>>> track the
> >>>>> widgets
> >> on a
> >>>>> per
> >>>>> tenant
> >> basis
> >>>>> and if
> >>>>> required
> >>>>> impose
> >>>>> limits on
> >>>>> the
> >> number
> >>>>> of
> >> widgets
> >>>>> that a
> >>>>> tenant
> >> may
> >>>>> consume
> >> at a
> >>>>> time,
> >> during
> >>>>> a course
> >> of
> >>>>> a period
> >> of
> >>>>> time, and
> >> so
> >>>>> on.
> >>>>>
> >>>>>
> >>>>> No, this last
> >> part
> >>>>> is absolutely not
> >>>>> what I think
> >> quota
> >>>>> management
> >>>>> should be about.
> >>>>>
> >>>>> Rate limiting --
> >>>>> i.e. how many
> >>>>> requests a
> >>>>> particular user
> >> can
> >>>>> make of
> >>>>> an API in a given
> >>>>> period of time --
> >>>>> should *not* be
> >>>>> handled by
> >>>>> OpenStack API
> >>>>> services, IMHO.
> >> It
> >>>>> is the
> >>>>> responsibility of
> >>>>> the
> >>>>> deployer to
> >> handle
> >>>>> this using
> >>>>> off-the-shelf
> >>>>> rate-limiting
> >>>>> solutions
> >>>>>
> >>>>>
> >>>>> (open source or proprietary).
> >>>>>
> >>>>>
> >>>>> Quotas should
> >> only
> >>>>> be about the hard
> >>>>> limit of
> >> different
> >>>>> types of
> >>>>> resources that a
> >>>>> user or group of
> >>>>> users can consume
> >> at
> >>>>> a given time.
> >>>>>
> >>>>>
> >>>>> [amrith] OK, good point.
> >>>>> Agreed as stipulated.
> >>>>>
> >>>>>
> >>>>> Such a
> >>>>>
> >> hypothetical
> >>>>> service
> >> may
> >>>>> also
> >> consume
> >>>>> resources
> >>>>> from
> >> other
> >>>>> services
> >>>>> that it
> >>>>> wishes to
> >>>>> track,
> >> and
> >>>>> impose
> >>>>> limits
> >> on.
> >>>>>
> >>>>> Yes, absolutely
> >>>>> agreed.
> >>>>>
> >>>>>
> >>>>> It is
> >> also
> >> understood
> >>>>> as Jay
> >> Pipes
> >>>>> points
> >> out
> >>>>> in [4]
> >> that
> >>>>> the
> >> actual
> >>>>> process
> >> of
> >> provisioning
> >>>>> widgets
> >>>>> could be
> >>>>> time
> >>>>> consuming
> >>>>> and it is
> >>>>>
> >> ill-advised
> >>>>> to hold a
> >>>>> database
> >>>>>
> >> transaction
> >>>>> of any
> >> kind
> >>>>> open for
> >>>>> that
> >>>>> duration
> >> of
> >>>>> time.
> >>>>> Ensuring
> >>>>> that a
> >> user
> >>>>> does not
> >>>>> exceed
> >> some
> >>>>> limit on
> >>>>> the
> >> number
> >>>>> of
> >>>>>
> >> concurrent
> >>>>> widgets
> >> that
> >>>>> he or she
> >>>>> may
> >> create
> >>>>> therefore
> >>>>> requires
> >>>>> some
> >>>>> mechanism
> >> to
> >>>>> track
> >>>>> in-flight
> >>>>> requests
> >> for
> >>>>> widgets.
> >> I
> >>>>> view
> >> these
> >>>>> as
> >> "intent"
> >>>>> but not
> >> yet
> >> materialized.
> >>>>>
> >>>>> It has nothing to
> >> do
> >>>>> with the amount
> >> of
> >>>>> concurrent
> >> widgets
> >>>>> that a
> >>>>> user can create.
> >>>>> It's just about
> >> the
> >>>>> total number of
> >> some
> >>>>> resource
> >>>>> that may be
> >> consumed
> >>>>> by that user.
> >>>>>
> >>>>> As for an
> >> "intent",
> >>>>> I don't believe
> >>>>> tracking intent
> >> is
> >>>>> the right way
> >>>>> to go at all. As
> >>>>> I've mentioned
> >>>>> before, the major
> >>>>> problem in Nova's
> >>>>> quota system is
> >> that
> >>>>> there are two
> >> tables
> >>>>> storing resource
> >>>>> usage
> >>>>> records: the
> >>>>> *actual* resource
> >>>>> usage tables (the
> >>>>> allocations table
> >> in
> >>>>> the new
> >>>>> resource-
> >> providers
> >>>>> modeling and the
> >>>>> instance_extra,
> >>>>> pci_devices and
> >>>>> instances table
> >> in
> >>>>> the legacy
> >> modeling)
> >>>>> and the *quota
> >>>>> usage* tables
> >>>>> (quota_usages and
> >>>>> reservations
> >>>>> tables). The
> >>>>> quota_usages
> >> table
> >>>>> does
> >>>>> not need to exist
> >> at
> >>>>> all, and neither
> >>>>> does the
> >>>>> reservations
> >> table.
> >>>>> Don't do
> >>>>> intent-based
> >>>>> consumption.
> >>>>> Instead, just
> >>>>> consume (claim)
> >> by
> >>>>> writing a record
> >> for
> >>>>> the resource
> >> class
> >>>>> consumed on a
> >>>>> provider into
> >>>>> the actual
> >> resource
> >>>>> usages table and
> >>>>> then "check
> >> quotas"
> >>>>> by querying
> >>>>> the *actual*
> >>>>> resource usages
> >> and
> >>>>> comparing the
> >>>>> SUM(used) values,
> >>>>> grouped by
> >> resource
> >>>>> class, against
> >> the
> >>>>> appropriate quota
> >>>>> limits for
> >>>>> the user. The
> >>>>> introduction of
> >> the
> >>>>> quota_usages and
> >>>>> reservations
> >>>>> tables to cache
> >>>>> usage records is
> >> the
> >>>>> primary reason
> >> for
> >>>>> the race
> >>>>> problems in the
> >> Nova
> >>>>> (and
> >>>>> other) quota
> >> system
> >>>>> because every
> >> time
> >>>>> you introduce a
> >>>>> caching system
> >>>>> for
> >> highly-volatile
> >>>>> data (like usage
> >>>>> records) you
> >>>>> introduce
> >>>>> complexity into
> >> the
> >>>>> write path and
> >> the
> >>>>> need to track the
> >>>>> same thing
> >>>>> across multiple
> >>>>> writes to
> >> different
> >>>>> tables
> >> needlessly.
> >>>>>
> >>>>> [amrith] I don't agree,
> >> I'll
> >>>>> respond to this and the
> >> next
> >>>>> comment group
> >>>>>
> >>>>>
> >>>>>
> >>>>> together. See below.
> >>>>>
> >>>>>
> >>>>> Looking
> >> up
> >>>>> at this
> >>>>> whole
> >>>>>
> >> infrastructure from the perspective of the
> >>>>> database,
> >> I
> >>>>> think we
> >>>>> should
> >>>>> require
> >> that
> >>>>> the
> >> database
> >>>>> must not
> >> be
> >>>>> required
> >> to
> >>>>> operate
> >> in
> >>>>> any
> >>>>> isolation
> >>>>> mode
> >> higher
> >>>>> than
> >>>>>
> >> READ-COMMITTED; more about that later (i.e. requiring a database run
> >>>>> either
> >>>>>
> >> serializable
> >>>>> or
> >>>>>
> >> repeatable
> >>>>> read is a
> >>>>> show
> >>>>> stopper).
> >>>>>
> >>>>>
> >>>>> This is an
> >>>>> implementation
> >>>>> detail is not
> >>>>> relevant to the
> >>>>> discussion
> >>>>> about what the
> >>>>> interface of a
> >> quota
> >>>>> library would
> >> look
> >>>>> like.
> >>>>>
> >>>>>
> >>>>> [amrith] I disagree, let
> >> me
> >>>>> give you an example of
> >> why.
> >>>>> Earlier, I wrote:
> >>>>>
> >>>>> Such a
> >>>>>
> >> hypothetical
> >>>>> service
> >> may
> >>>>> also
> >> consume
> >>>>> resources
> >>>>> from
> >> other
> >>>>> services
> >>>>> that it
> >>>>> wishes to
> >>>>> track,
> >> and
> >>>>> impose
> >>>>> limits
> >> on.
> >>>>>
> >>>>> And you responded:
> >>>>>
> >>>>>
> >>>>> Yes, absolutely
> >>>>> agreed.
> >>>>>
> >>>>>
> >>>>>
> >>>>> So let's take this
> >>>>> hypothetical service that
> >> in
> >>>>> response to a user
> >>>>>
> >>>>>
> >>>>>
> >>>>> request, will provision a Cinder
> >>>>> volume and a Nova instance. Let's
> >>>>> assume
> >>>>> that the service also imposes
> >> limits
> >>>>> on the number of cinder volumes
> >> and
> >>>>> nova instances the user may
> >>>>> provision; independent of limits
> >>>>> that Nova and
> >>>>> Cinder may themselves maintain.
> >>>>>
> >>>>> One way that the
> >>>>> hypothetical service can
> >>>>> function is this:
> >>>>>
> >>>>> (a) check Cinder quota,
> >> if
> >>>>> successful, create cinder
> >>>>> volume
> >>>>> (b) check Nova quota, if
> >>>>> successful, create nova
> >>>>> instance with cinder
> >>>>> volume attachment
> >>>>>
> >>>>> Now, this is sub-optimal
> >> as
> >>>>> there are going to be
> >> some
> >>>>> number of cases
> >>>>>
> >>>>>
> >>>>> where the nova quota check fails.
> >>>>> Now you have needlessly created
> >> and
> >>>>> will
> >>>>> have to release a cinder volume.
> >> It
> >>>>> also takes longer to fail.
> >>>>>
> >>>>> Another way to do this is
> >>>>> this:
> >>>>>
> >>>>> (1) check Cinder quota,
> >> if
> >>>>> successful, check Nova
> >>>>> quota, if successful
> >>>>> proceed to (2) else error
> >>>>> out
> >>>>> (2) create cinder volume
> >>>>> (3) create nova instance
> >>>>> with cinder attachment.
> >>>>>
> >>>>> I'm trying to get to this
> >>>>> latter form of doing
> >> things.
> >>>>> Easy, you might say ...
> >>>>> theoretically this should
> >>>>> simply be:
> >>>>>
> >>>>> BEGIN;
> >>>>> -- Get data to do the
> >> Cinder
> >>>>> check
> >>>>>
> >>>>> SELECT ......
> >>>>>
> >>>>> -- Do the cinder check
> >>>>>
> >>>>> INSERT INTO ....
> >>>>>
> >>>>> -- Get data to do the
> >> Nova
> >>>>> check
> >>>>>
> >>>>> SELECT ....
> >>>>>
> >>>>> -- Do the Nova check
> >>>>>
> >>>>> INSERT INTO ...
> >>>>>
> >>>>> COMMIT
> >>>>>
> >>>>> You can only make this
> >> work
> >>>>> if you ran at isolation
> >>>>> level serializable.
> >>>>>
> >>>>>
> >>>>> Why?
> >>>>>
> >>>>>
> >>>>> To make this run at
> >>>>> isolation level
> >>>>> REPEATABLE-READ, you must
> >>>>> enforce
> >>>>>
> >>>>>
> >>>>>
> >>>>> constraints at the database level
> >>>>> that will fail the commit. But
> >> wait,
> >>>>> you
> >>>>> can't do that because the data
> >> about
> >>>>> the global limits may not be in
> >> the
> >>>>> same database as the usage
> >> records.
> >>>>> Later you talk about caching and
> >>>>> stuff; all that doesn't help a
> >>>>> database constraint.
> >>>>>
> >>>>> For this reason, I think
> >>>>> there is going to have to
> >> be
> >>>>> some cognizance to
> >>>>>
> >>>>>
> >>>>>
> >>>>> the database isolation level in
> >> the
> >>>>> design of the library, and I
> >> think
> >>>>> it
> >>>>> will also impact the API that can
> >> be
> >>>>> constructed.
> >>>>>
> >>>>> In
> >> general
> >> therefore, I
> >>>>> believe
> >> that
> >>>>> the
> >>>>>
> >> hypothetical
> >>>>> service
> >>>>>
> >> processing
> >>>>> requests
> >> for
> >>>>> widgets
> >>>>> would
> >> have
> >>>>> to handle
> >>>>> three
> >> kinds
> >>>>> of
> >>>>>
> >> operations,
> >> provision,
> >>>>> modify,
> >> and
> >>>>> destroy.
> >> The
> >>>>> names
> >> are, I
> >>>>> believe,
> >>>>>
> >> self-explanatory.
> >>>>>
> >>>>> Generally,
> >>>>> modification of a
> >>>>> resource doesn't
> >>>>> come into play.
> >> The
> >>>>> primary exception
> >> to
> >>>>> this is for
> >>>>> transferring of
> >>>>> ownership of some
> >>>>>
> >>>>>
> >>>>> resource.
> >>>>>
> >>>>>
> >>>>> [amrith] Trove RESIZE is
> >> a
> >>>>> huge benefit for users
> >> and
> >>>>> while it may be a
> >>>>>
> >>>>>
> >>>>>
> >>>>> pain as you say, this is still a
> >>>>> very real benefit. Trove allows
> >> you
> >>>>> to
> >>>>> resize both your storage (resize
> >> the
> >>>>> cinder volume) and resize your
> >>>>> instance (change the flavor).
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Without
> >> loss
> >>>>> of
> >>>>>
> >> generality,
> >>>>> one can
> >> say
> >>>>> that all
> >>>>> three of
> >>>>> them must
> >>>>> validate
> >>>>> that the
> >>>>> operation
> >>>>> does not
> >>>>> violate
> >> some
> >>>>> limit (no
> >>>>> more
> >>>>> than X
> >>>>> widgets,
> >> no
> >>>>> fewer
> >> than X
> >>>>> widgets,
> >>>>> rates,
> >> and
> >>>>> so on).
> >>>>>
> >>>>>
> >>>>> No, only the
> >>>>> creation (and
> >> very
> >>>>> rarely the
> >>>>> modification)
> >> needs
> >>>>> any
> >>>>> validation that a
> >>>>> limit could been
> >>>>> violated.
> >> Destroying
> >>>>> a resource
> >>>>> never needs to be
> >>>>> checked for limit
> >>>>> violations.
> >>>>>
> >>>>>
> >>>>> [amrith] Well, if you are
> >>>>> going to create a volume
> >> of
> >>>>> 10GB and your
> >>>>>
> >>>>>
> >>>>>
> >>>>> limit is 100GB, resizing it to
> >> 200GB
> >>>>> should fail, I think.
> >>>>>
> >>>>>
> >>>>> Assuming
> >>>>> that the
> >>>>> service
> >>>>>
> >> provisions
> >>>>> resources
> >>>>> from
> >> other
> >>>>> services,
> >>>>> it is
> >> also
> >> conceivable
> >>>>> that
> >> limits
> >>>>> be
> >> imposed
> >>>>> on the
> >>>>> quantum
> >> of
> >>>>> those
> >>>>> services
> >>>>> consumed.
> >> In
> >>>>> practice,
> >> I
> >>>>> can
> >> imagine
> >>>>> a service
> >>>>> like
> >>>>> Trove
> >> using
> >>>>> the
> >>>>> Delimiter
> >>>>> project
> >> to
> >>>>> perform
> >> all
> >>>>> of these
> >>>>> kinds of
> >>>>> limit
> >>>>> checks;
> >> I'm
> >>>>> not
> >>>>>
> >> suggesting
> >>>>> that it
> >> does
> >>>>> this
> >> today,
> >>>>> nor that
> >>>>> there is
> >> an
> >>>>> immediate
> >>>>> plan to
> >>>>> implement
> >>>>> all of
> >> them,
> >>>>> just that
> >>>>> these
> >>>>> all seem
> >>>>> like good
> >>>>> uses a
> >> Quota
> >> Management
> >> capability.
> >>>>> - User
> >> may
> >>>>> not have
> >>>>> more than
> >> 25
> >>>>> database
> >>>>> instances
> >> at
> >>>>> a
> >>>>>
> >>>>>
> >>>>> time
> >>>>>
> >>>>>
> >>>>>
> >> -
> >>>>> User may
> >> not
> >>>>> have more
> >>>>> than 4
> >>>>> clusters
> >> at
> >>>>> a time
> >>>>> - User
> >> may
> >>>>> not
> >> consume
> >>>>> more than
> >>>>> 3TB of
> >> SSD
> >>>>> storage
> >> at a
> >>>>> time
> >>>>>
> >>>>>
> >>>>> Only if SSD
> >> storage
> >>>>> is a distinct
> >>>>> resource class
> >> from
> >>>>> DISK_GB. Right
> >>>>> now, Nova makes
> >> no
> >>>>> differentiation
> >>>>> w.r.t. SSD or HDD
> >> or
> >>>>> shared vs.
> >>>>> local block
> >> storage.
> >>>>>
> >>>>> [amrith] It matters not
> >> to
> >>>>> Trove whether Nova does
> >> nor
> >>>>> not. Cinder
> >>>>>
> >>>>>
> >>>>>
> >>>>> supports volume-types and users
> >> DO
> >>>>> want to limit based on
> >> volume-type
> >>>>> (for
> >>>>> example).
> >>>>>
> >>>>>
> >> -
> >>>>> User may
> >> not
> >>>>> launch
> >> more
> >>>>> than 10
> >> huge
> >>>>> instances
> >> at
> >>>>> a
> >>>>> time
> >>>>>
> >>>>>
> >>>>> What is the point
> >> of
> >>>>> such a limit?
> >>>>>
> >>>>>
> >>>>>
> >>>>> [amrith] Metering usage,
> >>>>> placing limitations on
> >> the
> >>>>> quantum of resources
> >>>>>
> >>>>>
> >>>>>
> >>>>> that a user may provision. Same
> >> as
> >>>>> with Nova. A flavor is merely a
> >>>>> simple
> >>>>> way to tie together a bag of
> >>>>> resources. It is a way to
> >> restrict
> >>>>> access,
> >>>>> for example, to specific
> >> resources
> >>>>> that are available in the cloud.
> >>>>> HUGE
> >>>>> is just an example I gave, pick
> >> any
> >>>>> flavor you want, and here's how a
> >>>>> service like Trove uses it.
> >>>>>
> >>>>> Users can ask to launch
> >> an
> >>>>> instance of a specific
> >>>>> database+version;
> >>>>>
> >>>>>
> >>>>>
> >>>>> MySQL 5.6-48 for example. Now, an
> >>>>> operator can restrict the
> >> instance
> >>>>> flavors, or volume types that can
> >> be
> >>>>> associated with the specific
> >>>>> datastore. And the flavor could
> >> be
> >>>>> used to map to, for example
> >> whether
> >>>>> the
> >>>>> instance is running on bare metal
> >> or
> >>>>> in a VM and if so with what kind
> >> of
> >>>>> hardware. That's a useful
> >> construct
> >>>>> for a service like Trove.
> >>>>>
> >>>>>
> >> -
> >>>>> User may
> >> not
> >>>>> launch
> >> more
> >>>>> than 3
> >>>>> clusters
> >> an
> >>>>> hour
> >>>>>
> >>>>>
> >>>>>
> >>>>> -1. This is rate
> >>>>> limiting and
> >> should
> >>>>> be handled by
> >>>>> rate-limiting
> >>>>>
> >>>>>
> >>>>>
> >>>>> services.
> >>>>>
> >>>>>
> >>>>>
> >> -
> >>>>> No more
> >> than
> >>>>> 500
> >> copies
> >>>>> of Oracle
> >>>>> may be
> >> run
> >>>>> at a time
> >>>>>
> >>>>>
> >>>>>
> >>>>> Is "Oracle" a
> >>>>> resource class?
> >>>>>
> >>>>>
> >>>>>
> >>>>> [amrith] As I view it,
> >> every
> >>>>> project should be free to
> >>>>> define its own
> >>>>>
> >>>>>
> >>>>>
> >>>>> set of resource classes and meter
> >>>>> them as it feels fit. So, while
> >>>>> Oracle
> >>>>> licenses may not, conceivably a
> >> lot
> >>>>> of things that Nova, Cinder, and
> >> the
> >>>>> other core projects don't care
> >>>>> about, are in fact relevant for a
> >>>>> consumer
> >>>>> of this library.
> >>>>>
> >>>>> While
> >> Nova
> >>>>> would be
> >> the
> >>>>> service
> >> that
> >>>>> limits
> >> the
> >>>>> number of
> >>>>> instances
> >>>>> a user
> >> can
> >>>>> have at a
> >>>>> time, the
> >>>>> ability
> >> for
> >>>>> a service
> >> to
> >>>>> limit
> >> this
> >>>>> further
> >>>>> should
> >> not
> >>>>> be
> >>>>>
> >> underestimated.
> >>>>> In turn,
> >>>>> should
> >> Nova
> >>>>> and
> >> Cinder
> >>>>> also use
> >> the
> >>>>> same
> >> Quota
> >> Management
> >>>>> Library,
> >>>>> they may
> >>>>> each
> >> impose
> >> limitations
> >>>>> like:
> >>>>>
> >>>>> - User
> >> may
> >>>>> not
> >> launch
> >>>>> more than
> >> 20
> >>>>> huge
> >>>>> instances
> >> at
> >>>>> a
> >>>>> time
> >>>>>
> >>>>>
> >>>>> Not a useful
> >>>>> limitation IMHO.
> >>>>>
> >>>>>
> >>>>>
> >>>>> [amrith] I beg to differ.
> >>>>> Again a huge instance is
> >>>>> just an example of
> >>>>>
> >>>>>
> >>>>>
> >>>>> some flavor; and the idea is to
> >>>>> allow a project to place its own
> >>>>> metrics
> >>>>> and meter based on those.
> >>>>>
> >>>>>
> >> -
> >>>>> User may
> >> not
> >>>>> launch
> >> more
> >>>>> than 3
> >>>>> instances
> >> in
> >>>>> a minute
> >>>>>
> >>>>>
> >>>>>
> >>>>> -1. This is rate
> >>>>> limiting.
> >>>>>
> >>>>>
> >>>>>
> >> -
> >>>>> User may
> >> not
> >>>>> consume
> >> more
> >>>>> than 15TB
> >> of
> >>>>> SSD at a
> >>>>> time
> >>>>> - User
> >> may
> >>>>> not have
> >>>>> more than
> >> 30
> >>>>> volumes
> >> at a
> >>>>> time
> >>>>>
> >>>>> Again,
> >> I'm
> >>>>> not
> >> implying
> >>>>> that
> >> either
> >>>>> Nova or
> >>>>> Cinder
> >>>>> should
> >>>>> provide
> >>>>> these
> >>>>>
> >> capabilities.
> >>>>> With this
> >> in
> >>>>> mind, I
> >>>>> believe
> >> that
> >>>>> the
> >> minimal
> >>>>> set of
> >>>>>
> >> operations
> >>>>> that
> >>>>> Delimiter
> >>>>> should
> >>>>> provide
> >> are:
> >>>>> -
> >>>>>
> >> define_resource(name, max, min, user_max, user_min, ...)
> >>>>>
> >>>>> What would the
> >> above
> >>>>> do? What service
> >>>>> would it be
> >> speaking
> >>>>> to?
> >>>>>
> >>>>>
> >>>>>
> >>>>> [amrith] I assume that
> >> this
> >>>>> would speak with some
> >>>>> backend (either
> >>>>>
> >>>>>
> >>>>>
> >>>>> keystone or the project itself)
> >> and
> >>>>> record these designated limits.
> >> This
> >>>>> is the way to register a project
> >>>>> specific metric like "Oracle
> >>>>> licenses".
> >>>>>
> >>>>>
> >> -
> >> update_resource_limits(name, user, user_max, user_min,
> >>>>> ...)
> >>>>>
> >>>>>
> >>>>> This doesn't
> >> belong
> >>>>> in a quota
> >> library.
> >>>>> It belongs as a
> >> REST
> >>>>> API in
> >>>>> Keystone.
> >>>>>
> >>>>>
> >>>>> [amrith] Fine, same place
> >>>>> where the previous thing
> >>>>> stores the global
> >>>>>
> >>>>>
> >>>>>
> >>>>> defaults is the target of this
> >> call.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >> -
> >> reserve_resource(name, user, size, parent_resource, ...)
> >>>>>
> >>>>>
> >>>>> This doesn't
> >> belong
> >>>>> in a quota
> >> library
> >>>>> at all. I think
> >>>>> reservations
> >>>>> are not germane
> >> to
> >>>>> resource
> >> consumption
> >>>>> and should be
> >>>>> handled by an
> >>>>> external service
> >> at
> >>>>> the orchestration
> >>>>> layer.
> >>>>>
> >>>>>
> >>>>> [amrith] Again not true,
> >> as
> >>>>> illustrated above this
> >>>>> library is the thing
> >>>>>
> >>>>>
> >>>>>
> >>>>> that projects could use to
> >> determine
> >>>>> whether or not to honor a
> >> request.
> >>>>> This reserve/provision process
> >> is, I
> >>>>> believe required because of the
> >>>>> vagaries of how we want to
> >> implement
> >>>>> this in the database.
> >>>>>
> >>>>>
> >> -
> >> provision_resource(resource, id)
> >>>>>
> >>>>>
> >>>>> A quota library
> >>>>> should not be
> >>>>> provisioning
> >>>>> anything. A quota
> >>>>> library
> >>>>> should simply
> >>>>> provide a
> >> consistent
> >>>>> interface for
> >>>>> *checking* that a
> >>>>> structured
> >> request
> >>>>> for some set of
> >>>>> resources *can*
> >> be
> >>>>> provided by the
> >>>>> service.
> >>>>>
> >>>>>
> >>>>> [amrith] This does not
> >>>>> actually call Nova or
> >>>>> anything; merely that
> >> AFTER
> >>>>>
> >>>>>
> >>>>> the hypothetical service has
> >> called
> >>>>> NOVA, this converts the
> >> reservation
> >>>>> (which can expire) into an actual
> >>>>> allocation.
> >>>>>
> >>>>>
> >> -
> >> update_resource(id or resource, newsize)
> >>>>>
> >>>>>
> >>>>> Resizing
> >> resources
> >>>>> is a bad idea,
> >> IMHO.
> >>>>> Resources are
> >> easier
> >>>>> to deal
> >>>>> with when they
> >> are
> >>>>> considered of
> >>>>> immutable size
> >> and
> >>>>> simple (i.e. not
> >>>>> complex or
> >> nested).
> >>>>> I think the
> >> problem
> >>>>> here is in the
> >>>>> definition of
> >>>>> resource classes
> >>>>> improperly.
> >>>>>
> >>>>>
> >>>>> [amrith] Let's leave the
> >>>>> quota library aside. This
> >>>>> assertion strikes at
> >>>>>
> >>>>>
> >>>>>
> >>>>> the very heart of things like
> >> Nova
> >>>>> resize, or for that matter Cinder
> >>>>> volume resize. Are those all bad
> >>>>> ideas? I made a 500GB Cinder
> >> volume
> >>>>> and
> >>>>> it is getting close to full. I'd
> >>>>> like to resize it to 2TB. Are you
> >>>>> saying
> >>>>> that's not a valid use case?
> >>>>>
> >>>>> For example, a
> >>>>> "cluster" is not
> >> a
> >>>>> resource. It is a
> >>>>> collection of
> >>>>> resources of type
> >>>>> node. "Resizing"
> >> a
> >>>>> cluster is a
> >>>>> misnomer, because
> >>>>> you aren't
> >> resizing
> >>>>> a resource at
> >> all.
> >>>>> Instead, you are
> >>>>> creating or
> >>>>> destroying
> >> resources
> >>>>> inside the
> >> cluster
> >>>>> (i.e. joining or
> >>>>> leaving
> >>>>>
> >>>>>
> >>>>> cluster nodes).
> >>>>>
> >>>>>
> >>>>> BTW, this is also
> >>>>> why the "resize
> >>>>> instance" API in
> >>>>> Nova is such a
> >>>>> giant pain in the
> >>>>> ass. It's
> >> attempting
> >>>>> to "modify" the
> >>>>> instance
> >>>>>
> >>>>>
> >>>>> "resource"
> >>>>>
> >>>>>
> >>>>> when the instance
> >>>>> isn't really the
> >>>>> resource at all.
> >> The
> >>>>> VCPU, RAM_MB,
> >>>>> DISK_GB, and PCI
> >>>>> devices are the
> >>>>> actual resources.
> >>>>> The instance is a
> >>>>> convenient way to
> >>>>> tie those
> >> resources
> >>>>> together, and
> >> doing
> >>>>> a "resize"
> >>>>> of the instance
> >>>>> behind the scenes
> >>>>> actually performs
> >> a
> >>>>> *move*
> >>>>> operation, which
> >>>>> isn't a *change*
> >> of
> >>>>> the original
> >>>>> resources.
> >> Rather,
> >>>>> it is a creation
> >> of
> >>>>> a new set of
> >>>>> resources (of the
> >>>>> new amounts) and
> >> a
> >>>>> deletion of the
> >> old
> >>>>> set of resources.
> >>>>>
> >>>>>
> >>>>> [amrith] that's fine, if
> >> all
> >>>>> we want is to handle the
> >>>>> resize operation
> >>>>>
> >>>>>
> >>>>>
> >>>>> as a new instance followed by a
> >>>>> deletion, that's great. But that
> >>>>> semantic
> >>>>> isn't necessarily the case for
> >>>>> something like (say) cinder.
> >>>>>
> >>>>> The "resize" API
> >>>>> call adds some
> >> nasty
> >>>>> confirmation and
> >>>>> cancel
> >>>>> semantics to the
> >>>>> calling interface
> >>>>> that hint that
> >> the
> >>>>> underlying
> >>>>> implementation of
> >>>>> the "resize"
> >>>>> operation is in
> >>>>> actuality not a
> >>>>> resize
> >>>>> at all, but
> >> rather a
> >> create-new-and-delete-old-resources operation.
> >>>>>
> >>>>> [amrith] And that isn't
> >>>>> germane to a quota
> >> library,
> >>>>> I don't think. What
> >>>>>
> >>>>>
> >>>>>
> >>>>> is, is this. Do we want to treat
> >> the
> >>>>> transient state when there are
> >> (for
> >>>>> example of Nova) two instances,
> >> one
> >>>>> of the new flavor and one of the
> >> old
> >>>>> flavor, or not. But, from the
> >>>>> perspective of a quota library, a
> >>>>> resize
> >>>>> operation is merely a reset of
> >> the
> >>>>> quota by the delta in the
> >> resource
> >>>>> consumed.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >> -
> >> release_resource(id or resource)
> >>>>> -
> >>>>>
> >> expire_reservations()
> >>>>>
> >>>>> I see no need to
> >>>>> have reservations
> >> in
> >>>>> the quota library
> >> at
> >>>>> all, as
> >>>>> mentioned above.
> >>>>>
> >>>>>
> >>>>> [amrith] Then I think the
> >>>>> quota library must
> >> require
> >>>>> that either (a) the
> >>>>>
> >>>>>
> >>>>>
> >>>>> underlying database runs
> >>>>> serializable or (b) database
> >>>>> constraints can be
> >>>>> used to enforce that at commit
> >> the
> >>>>> global limits are adhered to.
> >>>>>
> >>>>> As for your
> >> proposed
> >>>>> interface and
> >>>>> calling structure
> >>>>> below, I think a
> >>>>> much simpler
> >>>>> proposal would
> >> work
> >>>>> better. I'll work
> >> on
> >>>>> a cross-project
> >>>>> spec that
> >> describes
> >>>>> this simpler
> >>>>> proposal, but the
> >>>>> basics would be:
> >>>>>
> >>>>> 1) Have Keystone
> >>>>> store quota
> >>>>> information for
> >>>>> defaults (per
> >>>>> service
> >>>>> endpoint), for
> >>>>> tenants and for
> >>>>> users.
> >>>>>
> >>>>> Keystone would
> >> have
> >>>>> the set of
> >> canonical
> >>>>> resource class
> >>>>> names, and
> >>>>> each project,
> >> upon
> >>>>> handling a new
> >>>>> resource class,
> >>>>> would be
> >>>>> responsible for a
> >>>>> change submitted
> >> to
> >>>>> Keystone to add
> >> the
> >>>>> new resource
> >>>>>
> >>>>>
> >>>>> class code.
> >>>>>
> >>>>>
> >>>>> Straw man REST
> >> API:
> >>>>>
> >> GET /quotas/resource-classes
> >>>>> 200 OK
> >>>>> {
> >>>>>
> >> "resource_classes":
> >>>>> {
> >>>>> "compute.vcpu": {
> >>>>> "service":
> >>>>> "compute",
> >>>>> "code":
> >>>>> "compute.vcpu",
> >>>>> "description": "A
> >>>>> virtual CPU unit"
> >>>>> },
> >>>>> "compute.ram_mb":
> >> {
> >>>>> "service":
> >>>>> "compute",
> >>>>> "code":
> >>>>> "compute.ram_mb",
> >>>>> "description":
> >>>>> "Memory in
> >>>>> megabytes"
> >>>>> },
> >>>>> ...
> >>>>> "volume.disk_gb":
> >> {
> >>>>> "service":
> >> "volume",
> >>>>> "code":
> >>>>> "volume.disk_gb",
> >>>>> "description":
> >>>>> "Amount of disk
> >>>>> space in
> >> gigabytes"
> >>>>> },
> >>>>> ...
> >>>>> "database.count":
> >> {
> >>>>> "service":
> >>>>> "database",
> >>>>> "code":
> >>>>> "database.count",
> >>>>> "description":
> >>>>> "Number of
> >> database
> >>>>> instances"
> >>>>> }
> >>>>> }
> >>>>> }
> >>>>>
> >>>>>
> >>>>> [amrith] Well, a user is
> >>>>> allowed to have a certain
> >>>>> compute quota (which
> >>>>>
> >>>>>
> >>>>>
> >>>>> is shared by Nova and Trove) but
> >>>>> also a Trove quota. How would
> >> your
> >>>>> representation represent that?
> >>>>>
> >>>>> # Get the default
> >>>>> limits for new
> >>>>> users...
> >>>>>
> >> GET /quotas/defaults
> >>>>> 200 OK
> >>>>> {
> >>>>> "quotas": {
> >>>>> "compute.vcpu":
> >> 100,
> >>>>> "compute.ram_mb":
> >>>>> 32768,
> >>>>> "volume.disk_gb":
> >>>>> 1000,
> >>>>> "database.count":
> >> 25
> >>>>> }
> >>>>> }
> >>>>>
> >>>>> # Get a specific
> >>>>> user's limits...
> >>>>>
> >> GET /quotas/users/{UUID}
> >>>>> 200 OK
> >>>>> {
> >>>>> "quotas": {
> >>>>> "compute.vcpu":
> >> 100,
> >>>>> "compute.ram_mb":
> >>>>> 32768,
> >>>>> "volume.disk_gb":
> >>>>> 1000,
> >>>>> "database.count":
> >> 25
> >>>>> }
> >>>>> }
> >>>>>
> >>>>> # Get a tenant's
> >>>>> limits...
> >>>>>
> >> GET /quotas/tenants/{UUID}
> >>>>> 200 OK
> >>>>> {
> >>>>> "quotas": {
> >>>>> "compute.vcpu":
> >>>>> 1000,
> >>>>> "compute.ram_mb":
> >>>>> 327680,
> >>>>> "volume.disk_gb":
> >>>>> 10000,
> >>>>> "database.count":
> >>>>> 250
> >>>>> }
> >>>>> }
> >>>>>
> >>>>> 2) Have Delimiter
> >>>>> communicate with
> >> the
> >>>>> above proposed
> >> new
> >>>>> Keystone
> >>>>> REST API and
> >> package
> >>>>> up data into an
> >>>>>
> >> oslo.versioned_objects interface.
> >>>>> Clearly all of
> >> the
> >>>>> above can be
> >> heavily
> >>>>> cached both on
> >> the
> >>>>> server and
> >>>>> client side since
> >>>>> they rarely
> >> change
> >>>>> but are read
> >> often.
> >>>>>
> >>>>> [amrith] Caching on the
> >>>>> client won't save you
> >> from
> >>>>> oversubscription if
> >>>>>
> >>>>>
> >>>>>
> >>>>> you don't run serializable.
> >>>>>
> >>>>>
> >>>>> The Delimiter
> >>>>> library could be
> >>>>> used to provide a
> >>>>> calling interface
> >>>>> for service
> >> projects
> >>>>> to get a user's
> >>>>> limits for a set
> >> of
> >>>>> resource
> >>>>>
> >>>>>
> >>>>> classes:
> >>>>>
> >>>>>
> >>>>> (please excuse
> >>>>> wrongness, typos,
> >>>>> and other stuff
> >>>>> below, it's just
> >> a
> >>>>> straw- man not
> >>>>> production
> >> working
> >>>>> code...)
> >>>>>
> >>>>> # file:
> >>>>>
> >> delimiter/objects/limits.py
> >>>>> import
> >>>>>
> >> oslo.versioned_objects.base as ovo import
> >> oslo.versioned_objects.fields as ovo_fields
> >>>>>
> >>>>> class
> >>>>>
> >> ResourceLimit(ovo.VersionedObjectBase):
> >>>>> # 1.0: Initial
> >>>>> version
> >>>>> VERSION = '1.0'
> >>>>>
> >>>>> fields = {
> >>>>> 'resource_class':
> >>>>>
> >> ovo_fields.StringField(),
> >>>>> 'amount':
> >>>>>
> >> ovo_fields.IntegerField(),
> >>>>> }
> >>>>>
> >>>>>
> >>>>> class
> >>>>>
> >> ResourceLimitList(ovo.VersionedObjectBase):
> >>>>> # 1.0: Initial
> >>>>> version
> >>>>> VERSION = '1.0'
> >>>>>
> >>>>> fields = {
> >>>>> 'resources':
> >>>>>
> >> ListOfObjectsField(ResourceLimit),
> >>>>> }
> >>>>>
> >>>>>
> >> @cache_this_heavily
> >> @remotable_classmethod
> >>>>> def
> >>>>>
> >> get_all_by_user(cls,
> >>>>> user_uuid):
> >>>>> """Returns a
> >> Limits
> >>>>> object that tells
> >>>>> the caller what a
> >>>>> user's
> >>>>> absolute limits
> >> for
> >>>>> the set of
> >> resource
> >>>>> classes in the
> >>>>> system.
> >>>>> """
> >>>>> # Grab a keystone
> >>>>> client session
> >>>>> object and
> >> connect
> >>>>> to Keystone
> >>>>> ks =
> >>>>>
> >> ksclient.Session(...)
> >>>>> raw_limits =
> >>>>>
> >> ksclient.get_limits_by_user()
> >>>>> return
> >>>>>
> >> cls(resources=[ResourceLimit(**d) for d in raw_limits])
> >>>>> 3) Each service
> >>>>> project would be
> >>>>> responsible for
> >>>>> handling the
> >>>>> consumption of a
> >> set
> >>>>> of requested
> >>>>> resource amounts
> >> in
> >>>>> an atomic and
> >>>>>
> >>>>>
> >>>>> consistent way.
> >>>>>
> >>>>>
> >>>>> [amrith] This is where
> >> the
> >>>>> rubber meets the road.
> >> What
> >>>>> is that atomic
> >>>>>
> >>>>>
> >>>>>
> >>>>> and consistent way? And what
> >>>>> computing infrastructure do you
> >> need
> >>>>> to
> >>>>> deliver this?
> >>>>>
> >>>>> The Delimiter
> >>>>> library would
> >> return
> >>>>> the limits that
> >> the
> >>>>> service would
> >>>>> pre- check before
> >>>>> claiming the
> >>>>> resources and
> >> either
> >>>>> post-check after
> >>>>> claim or utilize
> >> a
> >> compare-and-update
> >>>>> technique with a
> >>>>>
> >> generation/timestamp
> >>>>> during claiming
> >> to
> >>>>> prevent race
> >>>>> conditions.
> >>>>>
> >>>>> For instance, in
> >>>>> Nova with the new
> >>>>> resource
> >> providers
> >>>>> database schema
> >>>>> and doing claims
> >> in
> >>>>> the scheduler (a
> >>>>> proposed change),
> >> we
> >>>>> might do
> >>>>> something to the
> >>>>> effect of:
> >>>>>
> >>>>> from delimiter
> >>>>> import objects as
> >>>>> delim_obj from
> >>>>> delimier import
> >>>>> exceptions as
> >>>>> delim_exc from
> >> nova
> >>>>> import objects as
> >>>>> nova_obj
> >>>>>
> >>>>> request =
> >>>>>
> >> nova_obj.RequestSpec.get_by_uuid(request_uuid)
> >>>>> requested =
> >>>>> request.resources
> >>>>> limits =
> >>>>>
> >> delim_obj.ResourceLimitList.get_all_by_user(user_uuid)
> >>>>> allocations =
> >>>>>
> >> nova_obj.AllocationList.get_all_by_user(user_uuid)
> >>>>> # Pre-check for
> >>>>> violations
> >>>>> for
> >> resource_class,
> >>>>> requested_amount
> >> in
> >> requested.items():
> >>>>> limit_idx =
> >>>>>
> >> limits.resources.index(resource_class)
> >>>>> resource_limit =
> >>>>>
> >> limits.resources[limit_idx].amount
> >>>>> alloc_idx =
> >>>>>
> >> allocations.resources.index(resource_class)
> >>>>> resource_used =
> >>>>>
> >> allocations.resources[alloc_idx]
> >>>>> if (resource_used
> >> +
> >> requested_amount)>
> >>>>> resource_limit:
> >>>>> raise
> >>>>>
> >> delim_exc.QuotaExceeded
> >>>>>
> >>>>> [amrith] Is the above
> >> code
> >>>>> run with some global
> >> mutex
> >>>>> to prevent that
> >>>>>
> >>>>>
> >>>>>
> >>>>> two people don't believe that
> >> they
> >>>>> are good on quota at the same
> >> time?
> >>>>>
> >>>>> # Do claims in
> >>>>> scheduler in an
> >>>>> atomic,
> >> consistent
> >>>>> fashion...
> >>>>> claims =
> >>>>>
> >> scheduler_client.claim_resources(request)
> >>>>>
> >>>>> [amrith] Yes, each
> >> 'atomic'
> >>>>> claim on a
> >> repeatable-read
> >>>>> database could
> >>>>>
> >>>>>
> >>>>>
> >>>>> result in oversubscription.
> >>>>>
> >>>>>
> >>>>> # Post-check for
> >>>>> violations
> >>>>> allocations =
> >>>>>
> >> nova_obj.AllocationList.get_all_by_user(user_uuid)
> >>>>> # allocations now
> >>>>> include the
> >> claimed
> >>>>> resources from
> >> the
> >>>>> scheduler
> >>>>>
> >>>>> for
> >> resource_class,
> >>>>> requested_amount
> >> in
> >> requested.items():
> >>>>> limit_idx =
> >>>>>
> >> limits.resources.index(resource_class)
> >>>>> resource_limit =
> >>>>>
> >> limits.resources[limit_idx].amount
> >>>>> alloc_idx =
> >>>>>
> >> allocations.resources.index(resource_class)
> >>>>> resource_used =
> >>>>>
> >> allocations.resources[alloc_idx]
> >>>>> if resource_used>
> >>>>> resource_limit:
> >>>>> # Delete the
> >>>>> allocation
> >> records
> >>>>> for the resources
> >>>>> just claimed
> >>>>>
> >> delete_resources(claims)
> >>>>> raise
> >>>>>
> >> delim_exc.QuotaExceeded
> >>>>>
> >>>>> [amrith] Again, two
> >> people
> >>>>> could drive through this
> >>>>> code and both of
> >>>>> them could fail :(
> >>>>>
> >>>>> 4) The only other
> >>>>> thing that would
> >>>>> need to be done
> >> for
> >>>>> a first go of
> >>>>> the Delimiter
> >>>>> library is some
> >>>>> event listener
> >> that
> >>>>> can listen for
> >>>>> changes to the
> >> quota
> >>>>> limits for a
> >>>>>
> >> user/tenant/default
> >>>>> in Keystone.
> >>>>> We'd want the
> >>>>> services to be
> >> able
> >>>>> notify someone if
> >> a
> >>>>> reduction in
> >>>>> quota results in
> >> an
> >>>>> overquota
> >> situation.
> >>>>> Anyway, that's my
> >>>>> idea. Keep the
> >>>>> Delimiter library
> >>>>> small and focused
> >>>>> on describing the
> >>>>> limits only, not
> >> on
> >>>>> the resource
> >>>>> allocations. Have
> >>>>> the Delimiter
> >>>>> library present a
> >>>>> versioned object
> >>>>> interface so the
> >>>>> interaction
> >> between
> >>>>> the data exposed
> >> by
> >>>>> the Keystone REST
> >>>>> API for
> >>>>> quotas can evolve
> >>>>> naturally and
> >>>>> smoothly over
> >> time.
> >>>>> Best,
> >>>>> -jay
> >>>>>
> >>>>> Let me
> >>>>>
> >> illustrate
> >>>>> the way I
> >>>>> see these
> >>>>> things
> >>>>> fitting
> >>>>> together.
> >> A
> >> hypothetical
> >>>>> Trove
> >> system
> >>>>> may be
> >> setup
> >>>>> as
> >> follows:
> >>>>> - No more
> >>>>> than 2000
> >>>>> database
> >>>>> instances
> >> in
> >>>>> total,
> >> 300
> >>>>> clusters
> >>>>>
> >>>>>
> >>>>> in
> >>>>>
> >>>>>
> >>>>>
> >>>>> total
> >>>>> - Users
> >> may
> >>>>> not
> >> launch
> >>>>> more than
> >> 25
> >>>>> database
> >>>>>
> >> instances,
> >>>>> or 4
> >>>>> clusters
> >>>>> - The
> >>>>>
> >> particular
> >>>>> user
> >>>>> 'amrith'
> >> is
> >>>>> limited
> >> to 2
> >>>>> databases
> >>>>> and
> >>>>>
> >>>>>
> >>>>> 1
> >>>>>
> >>>>>
> >>>>>
> >>>>> cluster
> >>>>> - No user
> >>>>> may
> >> consume
> >>>>> more than
> >>>>> 20TB of
> >>>>> storage
> >> at a
> >>>>> time
> >>>>> - No user
> >>>>> may
> >> consume
> >>>>> more than
> >>>>> 10GB of
> >>>>> memory at
> >> a
> >>>>> time
> >>>>>
> >>>>> At
> >> startup,
> >>>>> I believe
> >>>>> that the
> >>>>> system
> >> would
> >>>>> make the
> >>>>> following
> >>>>> sequence
> >> of
> >>>>> calls:
> >>>>>
> >>>>> -
> >>>>>
> >> define_resource(databaseInstance, 2000, 0, 25, 0, ...)
> >>>>> -
> >>>>>
> >> update_resource_limits(databaseInstance, amrith, 2, 0,
> >>>>>
> >>>>> ...)
> >>>>>
> >>>>>
> >>>>>
> >> -
> >> define_resource(databaseCluster, 300, 0, 4, 0, ...)
> >>>>> -
> >>>>>
> >> update_resource_limits(databaseCluster, amrith, 1, 0, ...)
> >>>>> -
> >>>>>
> >> define_resource(storage, -1, 0, 20TB, 0, ...)
> >>>>> -
> >>>>>
> >> define_resource(memory, -1, 0, 10GB, 0, ...)
> >>>>> Assume
> >> that
> >>>>> the user
> >>>>> john
> >> comes
> >>>>> along and
> >>>>> asks for
> >> a
> >>>>> cluster
> >> with
> >>>>> 4
> >>>>> nodes,
> >> 1TB
> >>>>> storage
> >> per
> >>>>> node and
> >>>>> each node
> >>>>> having
> >> 1GB
> >>>>> of
> >> memory,
> >>>>> the
> >>>>> system
> >> would
> >>>>> go
> >> through
> >>>>> the
> >>>>> following
> >>>>> sequence:
> >>>>>
> >>>>> -
> >>>>>
> >> reserve_resource(databaseCluster, john, 1, None)
> >>>>> o this
> >>>>> returns a
> >>>>>
> >> resourceID
> >>>>> (say
> >>>>>
> >> cluster-resource-
> >>>>>
> >>>>> ID)
> >>>>>
> >>>>>
> >>>>>
> >>>>> o
> >> the
> >>>>> cluster
> >>>>> instance
> >>>>> that it
> >>>>> reserves
> >>>>> counts
> >>>>>
> >>>>>
> >>>>>
> >>>>> against
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >> the
> >>>>> limit of
> >> 300
> >>>>> cluster
> >>>>> instances
> >> in
> >>>>> total, as
> >>>>> well
> >>>>>
> >>>>>
> >>>>>
> >>>>> as
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >> the 4
> >>>>> clusters
> >>>>> that john
> >>>>> can
> >>>>>
> >> provision.
> >>>>> If
> >> 'amrith'
> >>>>>
> >>>>>
> >>>>> had
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >> requested
> >>>>> it, that
> >>>>> would
> >> have
> >>>>> been
> >> counted
> >>>>> against
> >>>>>
> >>>>>
> >>>>>
> >>>>> the
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >> limit
> >>>>> of 2
> >>>>> clusters
> >> for
> >>>>> the user.
> >>>>>
> >>>>> -
> >>>>>
> >> reserve_resource(databaseInstance, john, 1,
> >> cluster-resource-id)
> >>>>> -
> >>>>>
> >> reserve_resource(databaseInstance, john, 1,
> >> cluster-resource-id)
> >>>>> -
> >>>>>
> >> reserve_resource(databaseInstance, john, 1,
> >> cluster-resource-id)
> >>>>> -
> >>>>>
> >> reserve_resource(databaseInstance, john, 1,
> >> cluster-resource-id)
> >>>>> o this
> >>>>> returns
> >> four
> >>>>> resource
> >>>>> id's,
> >> let's
> >>>>> say
> >>>>>
> >> instance-1-id, instance-2-id, instance-3-id,
> >> instance-4-id
> >>>>> o note
> >> that
> >>>>> each
> >>>>> instance
> >> is
> >>>>> that, an
> >>>>> instance
> >> by
> >>>>> itself.
> >> it
> >>>>> is
> >> therefore
> >>>>> not right
> >> to
> >>>>> consider
> >>>>> this
> >>>>>
> >>>>>
> >>>>> as
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >> equivalent
> >>>>> to a call
> >> to
> >> reserve_resource() with a
> >>>>>
> >>>>>
> >>>>> size
> >>>>>
> >>>>>
> >>>>>
> >>>>> of
> >> 4,
> >> especially
> >>>>> because
> >> each
> >>>>> instance
> >>>>> could
> >> later
> >>>>>
> >>>>>
> >>>>> be
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> tracked
> >> as
> >>>>> an
> >>>>>
> >> individual
> >>>>> Nova
> >>>>> instance.
> >>>>>
> >>>>> -
> >>>>>
> >> reserve_resource(storage, john, 1TB, instance-1-id)
> >>>>> -
> >>>>>
> >> reserve_resource(storage, john, 1TB, instance-2-id)
> >>>>> -
> >>>>>
> >> reserve_resource(storage, john, 1TB, instance-3-id)
> >>>>> -
> >>>>>
> >> reserve_resource(storage, john, 1TB, instance-4-id)
> >>>>> o each of
> >>>>> them
> >> returns
> >>>>> some
> >>>>>
> >> resourceID,
> >>>>> let's say
> >>>>>
> >>>>>
> >>>>> they
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> returned
> >>>>>
> >> cinder-1-id,
> >> cinder-2-id,
> >> cinder-3-id,
> >> cinder-4-id
> >>>>> o since
> >> the
> >>>>> storage
> >> of
> >>>>> 1TB is a
> >>>>> unit, it
> >> is
> >>>>> treated
> >>>>>
> >>>>>
> >>>>> as
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >> such.
> >>>>> In other
> >>>>> words,
> >> you
> >>>>> don't
> >> need
> >>>>> to invoke
> >>>>>
> >> reserve_resource 10^12 times, once per byte
> >>>>> allocated
> >>>>> :)
> >>>>>
> >>>>> -
> >>>>>
> >> reserve_resource(memory, john, 1GB, instance-1-id)
> >>>>> -
> >>>>>
> >> reserve_resource(memory, john, 1GB, instance-2-id)
> >>>>> -
> >>>>>
> >> reserve_resource(memory, john, 1GB, instance-3-id)
> >>>>> -
> >>>>>
> >> reserve_resource(memory, john, 1GB, instance-4-id)
> >>>>> o each of
> >>>>> these
> >> return
> >> something,
> >>>>> say
> >>>>>
> >> Dg4KBQcODAENBQEGBAcEDA, CgMJAg8FBQ8GDwgLBA8FAg,
> >> BAQJBwYMDwAIAA0DBAkNAg, AQMLDA4OAgEBCQ0MBAMGCA. I
> >>>>>
> >>>>> have
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >> made
> >>>>> up
> >> arbitrary
> >>>>> strings
> >> just
> >>>>> to
> >> highlight
> >>>>> that we
> >>>>> really
> >> don't
> >>>>> track
> >> these
> >>>>> anywhere
> >> so
> >>>>> we don't
> >>>>> care
> >>>>>
> >>>>>
> >>>>> about
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >> them.
> >>>>> If all
> >> this
> >>>>> works,
> >> then
> >>>>> the
> >> system
> >>>>> knows
> >> that
> >>>>> John's
> >>>>> request
> >> does
> >>>>> not
> >> violate
> >>>>> any
> >> quotas
> >>>>> that it
> >> can
> >>>>> enforce,
> >> it
> >>>>> can then
> >> go
> >>>>> ahead and
> >>>>> launch
> >> the
> >>>>> instances
> >>>>> (calling
> >>>>> Nova),
> >>>>> provision
> >>>>> storage,
> >> and
> >>>>> so on.
> >>>>>
> >>>>> The
> >> system
> >>>>> then goes
> >>>>> and
> >> creates
> >>>>> four
> >> Cinder
> >>>>> volumes,
> >>>>> these are
> >>>>>
> >> cinder-1-uuid, cinder-2-uuid, cinder-3-uuid, cinder-4-uuid.
> >>>>> It can
> >> then
> >>>>> go and
> >>>>> confirm
> >>>>> those
> >>>>>
> >> reservations.
> >>>>> -
> >>>>>
> >> provision_resource(cinder-1-id, cinder-1-uuid)
> >>>>> -
> >>>>>
> >> provision_resource(cinder-2-id, cinder-2-uuid)
> >>>>> -
> >>>>>
> >> provision_resource(cinder-3-id, cinder-3-uuid)
> >>>>> -
> >>>>>
> >> provision_resource(cinder-4-id, cinder-4-uuid)
> >>>>> It could
> >>>>> then go
> >> and
> >>>>> launch 4
> >>>>> nova
> >>>>> instances
> >>>>> and
> >>>>> similarly
> >>>>> provision
> >>>>> those
> >>>>>
> >> resources,
> >>>>> and so
> >> on.
> >>>>> This
> >> process
> >>>>> could
> >> take
> >>>>> some
> >> minutes
> >>>>> and
> >>>>> holding a
> >>>>> database
> >>>>>
> >> transaction
> >>>>> open for
> >>>>> this is
> >> the
> >>>>> issue
> >> that
> >>>>> Jay
> >>>>> brings up
> >> in
> >>>>> [4]. We
> >>>>> don't
> >> have
> >>>>> to in
> >> this
> >>>>> proposed
> >>>>> scheme.
> >>>>>
> >>>>> Since the
> >>>>> resources
> >>>>> are all
> >>>>>
> >> hierarchically linked through the
> >>>>> overall
> >>>>> cluster
> >> id,
> >>>>> when the
> >>>>> cluster
> >> is
> >>>>> setup, it
> >>>>> can
> >> finally
> >>>>> go and
> >>>>> provision
> >>>>> that:
> >>>>>
> >>>>> -
> >>>>>
> >> provision_resource(cluster-resource-id, cluster-uuid)
> >>>>> When
> >> Trove
> >>>>> is done
> >> with
> >>>>> some
> >>>>>
> >> individual
> >>>>> resource,
> >> it
> >>>>> can go
> >> and
> >>>>> release
> >> it.
> >>>>> Note that
> >>>>> I'm
> >> thinking
> >>>>> this will
> >>>>> invoke
> >>>>>
> >> release_resource
> >>>>> with the
> >> ID
> >>>>> of the
> >>>>>
> >> underlying
> >>>>> object OR
> >>>>> the
> >>>>> resource.
> >>>>>
> >>>>> -
> >>>>>
> >> release_resource(cinder-4-id), and
> >>>>> -
> >>>>>
> >> release_resource(cinder-4-uuid)
> >>>>> are
> >>>>> therefore
> >>>>> identical
> >>>>> and
> >> indicate
> >>>>> that the
> >> 4th
> >>>>> 1TB
> >> volume
> >>>>> is now
> >>>>> released.
> >>>>> How this
> >>>>> will be
> >>>>>
> >> implemented
> >>>>> in
> >> Python,
> >>>>> kwargs or
> >>>>> some
> >>>>> other
> >>>>> mechanism
> >>>>> is, I
> >>>>> believe,
> >> an
> >> implementation detail.
> >>>>> Finally,
> >> it
> >>>>> releases
> >> the
> >>>>> cluster
> >>>>> resource
> >> by
> >>>>> doing
> >> this:
> >>>>> -
> >>>>>
> >> release_resource(cluster-resource-id)
> >>>>> This
> >> would
> >>>>> release
> >> the
> >>>>> cluster
> >> and
> >>>>> all
> >>>>> dependent
> >>>>> resources
> >> in
> >>>>> a
> >>>>> single
> >>>>>
> >> operation.
> >>>>> A user
> >> may
> >>>>> wish to
> >>>>> manage a
> >>>>> resource
> >>>>> that was
> >>>>>
> >> provisioned
> >>>>> from the
> >>>>> service.
> >>>>> Assume
> >> that
> >>>>> this
> >> results
> >>>>> in a
> >>>>> resizing
> >> of
> >>>>> the
> >>>>>
> >> instances,
> >>>>> then it
> >> is a
> >>>>> matter of
> >>>>> updating
> >>>>> that
> >>>>> resource.
> >>>>>
> >>>>> Assume
> >> that
> >>>>> the third
> >>>>> 1TB
> >> volume
> >>>>> is being
> >>>>> resized
> >> to
> >>>>> 2TB, then
> >> it
> >>>>> is
> >>>>> merely a
> >>>>> matter of
> >>>>> invoking:
> >>>>>
> >>>>> -
> >>>>>
> >> update_resource(cinder-3-uuid, 2TB)
> >>>>> Delimiter
> >>>>> can go
> >>>>> figure
> >> out
> >>>>> that
> >>>>>
> >> cinder-3-uuid is a 1TB device and
> >>>>> therefore
> >>>>> this is
> >> an
> >>>>> increase
> >> of
> >>>>> 1TB and
> >>>>> verify
> >> that
> >>>>> this is
> >>>>> within
> >>>>> the
> >> quotas
> >>>>> allowed
> >> for
> >>>>> the user.
> >>>>>
> >>>>> The thing
> >>>>> that I
> >> find
> >> attractive
> >>>>> about
> >> this
> >>>>> model of
> >>>>>
> >> maintaining
> >>>>> a
> >>>>> hierarchy
> >> of
> >> reservations
> >>>>> is that
> >> in
> >>>>> the event
> >> of
> >>>>> an error,
> >>>>> the
> >>>>> service
> >> need
> >>>>> merely
> >> call
> >> release_resource() on the highest level
> >> reservation
> >>>>> and the
> >>>>> Delimiter
> >>>>> project
> >> can
> >>>>> walk down
> >>>>> the chain
> >>>>> and
> >>>>> release
> >> all
> >>>>> the
> >>>>> resources
> >> or
> >> reservations
> >>>>> as
> >>>>>
> >> appropriate.
> >>>>> Under the
> >>>>> covers I
> >>>>> believe
> >> that
> >>>>> each of
> >>>>> these
> >>>>>
> >> operations
> >>>>> should be
> >>>>> atomic
> >> and
> >>>>> may
> >> update
> >>>>> multiple
> >>>>> database
> >>>>> tables
> >> but
> >>>>> these
> >> will
> >>>>> all be
> >>>>> short
> >> lived
> >> operations.
> >>>>> For
> >> example,
> >>>>> reserving
> >> an
> >>>>> instance
> >>>>> resource
> >>>>> would
> >>>>> increment
> >>>>> the
> >>>>> number of
> >>>>> instances
> >>>>> for the
> >> user
> >>>>> as well
> >> as
> >>>>> the
> >> number
> >>>>> of
> >> instances
> >>>>> on the
> >>>>> whole,
> >> and
> >>>>> this
> >> would
> >>>>> be an
> >> atomic
> >> operation.
> >>>>> I have
> >> two
> >>>>> primary
> >>>>> areas of
> >>>>> concern
> >>>>> about the
> >>>>> proposal
> >>>>> [3].
> >>>>>
> >>>>> The first
> >> is
> >>>>> that it
> >>>>> makes the
> >>>>> implicit
> >>>>>
> >> assumption
> >>>>> that the
> >>>>> "flat
> >> mode"
> >>>>> is
> >>>>>
> >> implemented.
> >>>>> That
> >>>>> provides
> >>>>> value to
> >> a
> >>>>>
> >>>>> consumer
> >>>>>
> >>>>>
> >>>>>
> >>>>> but I
> >> think
> >>>>> it leaves
> >> a
> >>>>> lot for
> >> the
> >>>>> consumer
> >> to
> >>>>> do. For
> >>>>>
> >>>>>
> >>>>>
> >>>>> example,
> >>>>>
> >>>>>
> >>>>>
> >> I
> >>>>> find it
> >> hard
> >>>>> to see
> >> how
> >>>>> the model
> >>>>> proposed
> >>>>> would
> >> handle
> >>>>>
> >>>>>
> >>>>> the
> >>>>>
> >>>>>
> >>>>>
> >>>>> release
> >> of
> >>>>> quotas,
> >>>>> leave
> >> alone
> >>>>> the case
> >> of
> >>>>> a nested
> >>>>> release
> >> of
> >>>>>
> >>>>> a
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >> hierarchy
> >>>>> of
> >>>>>
> >> resources.
> >>>>> The other
> >> is
> >>>>> the
> >> notion
> >>>>> that the
> >>>>>
> >> implementation will begin a
> >> transaction,
> >>>>> perform a
> >>>>> query(),
> >>>>> make some
> >>>>>
> >> manipulations, and
> >>>>> then do a
> >>>>> save().
> >> This
> >>>>> makes for
> >> an
> >> interesting
> >> transaction
> >> management
> >>>>> challenge
> >> as
> >>>>> it would
> >>>>> require
> >> the
> >> underlying
> >>>>>
> >>>>> database
> >>>>>
> >>>>>
> >>>>>
> >>>>> to run
> >> in
> >>>>> an
> >> isolation
> >>>>> mode of
> >> at
> >>>>> least
> >>>>>
> >> repeatable
> >>>>> reads and
> >>>>> maybe
> >> even
> >> serializable
> >>>>> which
> >> would
> >>>>> be a
> >>>>>
> >> performance
> >>>>> bear on
> >>>>>
> >>>>>
> >>>>> a
> >>>>>
> >>>>>
> >>>>>
> >>>>> heavily
> >>>>> loaded
> >>>>> system.
> >> If
> >>>>> run in
> >> the
> >> traditional
> >>>>> read-
> >>>>>
> >>>>>
> >>>>>
> >>>>> committed
> >>>>>
> >>>>>
> >>>>>
> >>>>> mode,
> >> this
> >>>>> would
> >>>>> silently
> >>>>> lead to
> >> over
> >> subscriptions, and
> >>>>>
> >>>>>
> >>>>> the
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >> violation
> >>>>> of quota
> >>>>> limits.
> >>>>>
> >>>>> I believe
> >>>>> that it
> >>>>> should be
> >> a
> >> requirement
> >>>>> that the
> >>>>> Delimiter
> >>>>> library
> >>>>> should be
> >>>>> able to
> >> run
> >>>>> against a
> >>>>> database
> >>>>> that
> >>>>> supports,
> >>>>> and is
> >>>>>
> >> configured
> >>>>> for
> >>>>>
> >> READ-COMMITTED, and should not require anything higher.
> >>>>> The model
> >>>>> proposed
> >>>>> above can
> >>>>> certainly
> >> be
> >> implemented
> >>>>> with a
> >>>>> database
> >>>>> running
> >>>>>
> >> READ-COMMITTED, and I believe that this is also
> >>>>> true with
> >>>>> the
> >> caveat
> >>>>> that the
> >>>>>
> >> operations
> >>>>> will be
> >>>>> performed
> >>>>> through
> >>>>>
> >>>>>
> >>>>> SQLAlchemy.
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> -amrith
> >>>>>
> >>>>> [1]
> >>>>>
> >> http://openstack.markmail.org/thread/tkl2jcyvzgifniux
> >>>>> [2]
> >>>>>
> >> http://openstack.markmail.org/thread/3cr7hoeqjmgyle2j
> >>>>> [3]
> >>>>>
> >> https://review.openstack.org/#/c/284454/
> >>>>> [4]
> >>>>>
> >> http://markmail.org/message/7ixvezcsj3uyiro6
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >> ____________________________________________________________________
> >>>>> __ ____
> >>>>> OpenStack
> >>>>>
> >> Development
> >>>>> Mailing
> >> List
> >>>>> (not for
> >>>>> usage
> >>>>>
> >> questions)
> >> Unsubscribe:
> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>
> >>>>>
> >> _____________________________________________________________________
> >>>>> _____ OpenStack
> >>>>> Development
> >> Mailing
> >>>>> List (not for
> >> usage
> >>>>> questions)
> >>>>> Unsubscribe:
> >>>>>
> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>
> >>>>>
> >> ______________________________________________________________________
> >>>>> ____ OpenStack
> >> Development
> >>>>> Mailing List (not for
> >> usage
> >>>>> questions)
> >>>>> Unsubscribe:
> >>>>>
> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>
> >>>>>
> >>
> __________________________________________________________________________
> >>>>> OpenStack Development Mailing
> >> List
> >>>>> (not for usage questions)
> >>>>> Unsubscribe:
> >>>>>
> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>
> >>>>>
> >>>>>
> >>
> __________________________________________________________________________
> >>>>> OpenStack Development Mailing List (not
> >> for
> >>>>> usage questions)
> >>>>> Unsubscribe:
> >>>>>
> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>
> >>>>>
> >>>>>
> >>
> __________________________________________________________________________
> >>>>> OpenStack Development Mailing List (not for usage
> >>>>> questions)
> >>>>> Unsubscribe:
> >>>>>
> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>
> __________________________________________________________________________
> >>>>> OpenStack Development Mailing List (not for usage questions)
> >>>>>
> >>>>> Unsubscribe:
> >>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>>>>
> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>
> >>>>
> >>
> __________________________________________________________________________
> >>>> OpenStack Development Mailing List (not for usage questions)
> >>>> Unsubscribe:
> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> __________________________________________________________________________
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: OpenStack-dev-
> request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-
> request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
More information about the OpenStack-dev
mailing list