Open Stack

Huge +1 from me on bringing sanity to the various quota implementations 
in OpenStack.  This is something we discussed back in Paris, but the 
conclusion at the time was that it wasn't practical for it to live in 
Oslo because there was no place for Oslo to store the data (Oslo libs 
dictating db schemas to consuming projects would get messy fast).  Now 
that Keystone is providing a central storage location we should be able 
to make more progress.

On 03/12/2018 11:45 AM, Lance Bragstad wrote:
> I missed the document describing the process for this sort of thing [0].
> So I'm back tracking a bit to go through a more formal process.
> 
> [0]
> http://specs.openstack.org/openstack/oslo-specs/specs/policy/new-libraries.html
> 
> # Proposed new library oslo.limit
> 
> This is a proposal to create a new library dedicated to enabling more
> consistent quota and limit enforcement across OpenStack.
> 
> ## Proposed library mission
> 
> Enforcing quotas and limits across OpenStack has traditionally been a
> tough problem to solve. Determining enforcement requires quota knowledge
> from the service along with information about the project owning the
> resource. Up until the Queens release, quota calculation and enforcement
> has been left to the services to implement, forcing them to understand
> complexities of keystone project structure. During the Pike and Queens
> PTG, there were several productive discussions towards redesigning the
> current approach to quota enforcement. Because keystone is the authority
> of project structure, it makes sense to allow keystone to hold the
> association between a resource limit and a project. This means services
> still need to calculate quota and usage, but the problem should be
> easier for services to implement since developers shouldn't need to
> re-implement possible hierarchies of projects and their associated
> limits. Instead, we can offload some of that work to a common library
> for services to consume that handles enforcing quota calculation based
> on limits associated to projects in keystone. This proposal is to have a
> new library called oslo.limit that fills that need.
> 
> ## Consuming projects
> 
> The services consuming this work will be any service that currently
> implements a quota system, or plans to implement one. Since keystone
> already supports unified limits and association of limits to projects,
> the implementation for consuming projects is easier. instead of having
> to re-write that implementation, developers need to ensure quota
> calculation to passed to the oslo.limit library somewhere in the API's
> validation layer. The pattern described here is very similar to the
> pattern currently used by services that leverage oslo.policy for
> authorization decisions.
> 
> ## Alternative libraries
> 
> It looks like there was an existing library that attempted to solve some
> of these problems, called delimiter [1]. It looks like delimiter could
> be used to talk to keystone about quota enforcement, where as the
> existing approach with oslo.limit would be to use keystone directly.
> Someone more familiar with the library (harlowja?) can probably shed
> more light on it's intended uses (I couldn't find much documentation),
> but the presentation linked in a previous note was helpful.
> 
> [1] https://github.com/openstack/delimiter

I took a look at delimiter as well.  There are a couple of points that I 
think justify a new library:

1) Delimiter appears to implement a pluggable backend strategy, which 
isn't what we're planning for oslo.limit.  We'll only be targeting 
Keystone as the storage backend.

2) Given that, it makes sense for the library to live in the oslo 
namespace.  Projects like Delimiter that live outside the oslo namespace 
are generally supposed to be usable without OpenStack, but a hard 
dependency on Keystone makes that impossible.  Given that we aren't 
overstocked with developers these days, I'd rather not try to solve 
quota problems for the whole world.  If at some point we get oslo.limit 
to a good place and can factor some bits out to delimiter then great, 
but I'd rather solve quota for just OpenStack first at this point.

3) It doesn't look like delimiter ever got to the point where it had 
adoption in OpenStack.  It isn't present in global-requirements that I 
can see and it's not clear to me from looking at the repo whether it 
even could be used in its current state.  In the end I think it might be 
just as much work to get delimiter to a point where it could be used and 
we'd still be left with some less-than-ideal design points.

> 
> ## Proposed adoption model/plan
> 
> The unified limit API [2] in keystone is currently marked as
> experimental, but the keystone team is actively collecting and
> addressing feedback that will result in stabilizing the API.
> Stabilization changes that effect the oslo.limit library will also be
> addressed before version 1.0.0 is released. From there, we can look to
> incorporate the library into various services that either have an
> existing quota implementation, or services that have a quota requirement
> but no implementation.
> 
> This should help us refine the interfaces between services and
> oslo.limit, while providing a facade to handle complexities of project
> hierarchies. This should enable adoption by simplifying the process and
> making it easier for quota to be implemented in a consistent way across
> services.
> 
> [2]
> https://docs.openstack.org/keystone/latest/admin/identity-unified-limits.html
> 
> ## Reviewer activity
> 
> At first thought, it makes sense to model the reviewer structure after
> the oslo.policy library, where the core team consists of people not only
> interested in limits and quota, but also people familiar with the
> keystone implementation of the unified limits API.

+1

> 
> ## Implementation
> 
> ### Primary Authors:
> 
>    Lance Bragstad (lbragstad at gmail.com) lbragstad
>    You?
> 
> ### Other contributors:
> 
>    You?
> 
> ## Work Items
> 
> * Create a new library called oslo.limit
> * Create a core group for the project
> * Define the minimum we need to enforce quota calculations in oslo.limit
> * Propose an implementation that allows services to test out quota
> enforcement via unified limits
> 
> ## References
> 
> Rocky PTG Etherpad for unified limits:
> https://etherpad.openstack.org/p/unified-limits-rocky-ptg
> 
> ## Revision History
> 
> Introduced in Rocky
> 
> 
> On 03/07/2018 11:55 PM, Joshua Harlow wrote:
>> So the following was a prior effort:
>>
>> https://github.com/openstack/delimiter
>>
>> Maybe just continue down the path of that and/or take that whole repo
>> over and iterate (or adjust the prior code, or ...)?? Or if not that's
>> ok to, ya'll get to decide.
>>
>> https://www.slideshare.net/vilobh/delimiter-openstack-cross-project-quota-library-proposal
>>
>>
>> Lance Bragstad wrote:
>>> Hi all,
>>>
>>> Per the identity-integration track at the PTG [0], I proposed a new oslo
>>> library for services to use for hierarchical quota enforcement [1]. Let
>>> me know if you have any questions or concerns about the library. If the
>>> oslo team would like, I can add an agenda item for next weeks oslo
>>> meeting to discuss.
>>>
>>> Thanks,
>>>
>>> Lance
>>>
>>> [0] https://etherpad.openstack.org/p/unified-limits-rocky-ptg
>>> [1] https://review.openstack.org/#/c/550491/
>>>
>>>
>>>
>>> __________________________________________________________________________
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>