Open Stack

Hi,

Would the new library support quota on domain level also? As it stands in
oslo-incubator it only does quota enforcement on project level only. The
use case for this is quota enforcement across multiple projects. For e.g.
as a cloud provider, I would like my customer to create only #X volumes
across all his projects.

-Ajaya

Cheers,
Ajaya

On Wed, Oct 15, 2014 at 7:04 PM, Doug Hellmann <doug at doughellmann.com>
wrote:

> Sigh. Because I typed the wrong command. Thanks for pointing that out.
>
> I don’t see any instances of “quota” in openstack-common.conf files:
>
> $ grep quota */openstack-common.conf
>
> or in any projects under "openstack/“:
>
> $ ls */*/openstack/common/quota.py
> ls: cannot access */*/openstack/common/quota.py: No such file or directory
>
> I don’t know where manila’s copy came from, but if it has been copied from
> the incubator by hand and then changed we should fix that up.
>
> Doug
>
> On Oct 15, 2014, at 5:28 AM, Valeriy Ponomaryov <vponomaryov at mirantis.com>
> wrote:
>
> But why "policy" is being discussed on "quota" thread?
>
> On Wed, Oct 15, 2014 at 11:55 AM, Valeriy Ponomaryov <
> vponomaryov at mirantis.com> wrote:
>
>> Manila project does use "policy" common code from incubator.
>>
>> Our "small" wrapper for it:
>> https://github.com/openstack/manila/blob/8203c51081680a7a9dba30ae02d7c43d6e18a124/manila/policy.py
>>
>>
>> On Wed, Oct 15, 2014 at 2:41 AM, Salvatore Orlando <sorlando at nicira.com>
>> wrote:
>>
>>> Doug,
>>>
>>> I totally agree with your findings on the policy module.
>>> Neutron already has some "customizations" there and we already have a
>>> few contributors working on syncing it back with oslo-incubator during the
>>> Kilo release cycle.
>>>
>>> However, my query was about the quota module.
>>> From what I gather it seems not a lot of projects use it:
>>>
>>> $ find . -name openstack-common.conf | xargs grep quota
>>> $
>>>
>>> Salvatore
>>>
>>> On 15 October 2014 00:34, Doug Hellmann <doug at doughellmann.com> wrote:
>>>
>>>>
>>>> On Oct 14, 2014, at 12:31 PM, Salvatore Orlando <sorlando at nicira.com>
>>>> wrote:
>>>>
>>>> Hi Doug,
>>>>
>>>> do you know if the existing quota oslo-incubator module has already
>>>> some active consumers?
>>>> In the meanwhile I've pushed a spec to neutron-specs for improving
>>>> quota management there [1]
>>>>
>>>>
>>>> It looks like a lot of projects are syncing the module:
>>>>
>>>> $ grep policy */openstack-common.conf
>>>>
>>>> barbican/openstack-common.conf:modules=gettextutils,jsonutils,log,local,timeutils,importutils,policy
>>>> ceilometer/openstack-common.conf:module=policy
>>>> cinder/openstack-common.conf:module=policy
>>>> designate/openstack-common.conf:module=policy
>>>> gantt/openstack-common.conf:module=policy
>>>> glance/openstack-common.conf:module=policy
>>>> heat/openstack-common.conf:module=policy
>>>> horizon/openstack-common.conf:module=policy
>>>> ironic/openstack-common.conf:module=policy
>>>> keystone/openstack-common.conf:module=policy
>>>> manila/openstack-common.conf:module=policy
>>>> neutron/openstack-common.conf:module=policy
>>>> nova/openstack-common.conf:module=policy
>>>> trove/openstack-common.conf:module=policy
>>>> tuskar/openstack-common.conf:module=policy
>>>>
>>>> I’m not sure how many are actively using it, but I wouldn’t expect them
>>>> to copy it in if they weren’t using it at all.
>>>>
>>>>
>>>> Now, I can either work on the oslo-incubator module and leverage it in
>>>> Neutron, or develop the quota module in Neutron, and move it to
>>>> oslo-incubator once we validate it with Neutron. The latter approach seems
>>>> easier from a workflow perspective - as it avoid the intermediate steps of
>>>> moving code from oslo-incubator to neutron. On the other hand it will delay
>>>> adoption in oslo-incubator.
>>>>
>>>>
>>>> The policy module is up for graduation this cycle. It may end up in its
>>>> own library, to allow us to build a review team for the code more easily
>>>> than if we put it in with some of the other semi-related modules like the
>>>> server code. We’re still working that out [1], and if you expect to make a
>>>> lot of incompatible changes we should delay graduation to make that simpler.
>>>>
>>>> Either way, since we have so many consumers, I think it would be easier
>>>> to have the work happen in Oslo somewhere so we can ensure those changes
>>>> are useful to and usable by all of the existing consumers.
>>>>
>>>> Doug
>>>>
>>>> [1] https://etherpad.openstack.org/p/kilo-oslo-library-proposals
>>>>
>>>>
>>>> What's your opinion?
>>>>
>>>> Regards,
>>>> Salvatore
>>>>
>>>> [1] https://review.openstack.org/#/c/128318/
>>>>
>>>> On 8 October 2014 18:52, Doug Hellmann <doug at doughellmann.com> wrote:
>>>>
>>>>>
>>>>> On Oct 8, 2014, at 7:03 AM, Davanum Srinivas <davanum at gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Salvatore, Joe,
>>>>> >
>>>>> > We do have this at the moment:
>>>>> >
>>>>> >
>>>>> https://github.com/openstack/oslo-incubator/blob/master/openstack/common/quota.py
>>>>> >
>>>>> > — dims
>>>>>
>>>>> If someone wants to drive creating a useful library during kilo,
>>>>> please consider adding the topic to the etherpad we’re using to plan summit
>>>>> sessions and then come participate in the Oslo meeting this Friday 16:00
>>>>> UTC.
>>>>>
>>>>> https://etherpad.openstack.org/p/kilo-oslo-summit-topics
>>>>>
>>>>> Doug
>>>>>
>>>>> >
>>>>> > On Wed, Oct 8, 2014 at 2:29 AM, Salvatore Orlando <
>>>>> sorlando at nicira.com> wrote:
>>>>> >>
>>>>> >> On 8 October 2014 04:13, Joe Gordon <joe.gordon0 at gmail.com> wrote:
>>>>> >>>
>>>>> >>>
>>>>> >>> On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg
>>>>> >>> <morgan.fainberg at gmail.com> wrote:
>>>>> >>>>
>>>>> >>>> Keeping the enforcement local (same way policy works today) helps
>>>>> limit
>>>>> >>>> the fragility, big +1 there.
>>>>> >>>>
>>>>> >>>> I also agree with Vish, we need a uniform way to talk about quota
>>>>> >>>> enforcement similar to how we have a uniform policy language /
>>>>> enforcement
>>>>> >>>> model (yes I know it's not perfect, but it's far closer to
>>>>> uniform than
>>>>> >>>> quota management is).
>>>>> >>>
>>>>> >>>
>>>>> >>> It sounds like maybe we should have an oslo library for quotas?
>>>>> Somewhere
>>>>> >>> where we can share the code,but keep the operations local to each
>>>>> service.
>>>>> >>
>>>>> >>
>>>>> >> This is what I had in mind as well. A simple library for quota
>>>>> enforcement
>>>>> >> which can be used regardless of where and how you do it, which
>>>>> might depend
>>>>> >> on the application business logic, the WSGI framework in use, or
>>>>> other
>>>>> >> factors.
>>>>> >>
>>>>> >>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> If there is still interest of placing quota in keystone, let's
>>>>> talk about
>>>>> >>>> how that will work and what will be needed from Keystone . The
>>>>> previous
>>>>> >>>> attempt didn't get much traction and stalled out early in
>>>>> implementation. If
>>>>> >>>> we want to revisit this lets make sure we have the resources
>>>>> needed and
>>>>> >>>> spec(s) in progress / info on etherpads (similar to how the
>>>>> multitenancy
>>>>> >>>> stuff was handled at the last summit) as early as possible.
>>>>> >>>
>>>>> >>>
>>>>> >>> Why not centralize quota management via the
>>>>> python-openstackclient, what
>>>>> >>> is the benefit of getting keystone involved?
>>>>> >>
>>>>> >>
>>>>> >> Providing this through the openstack client in my opinion has the
>>>>> >> disadvantage that users which either use the REST API direct or
>>>>> write their
>>>>> >> own clients won't leverage it. I don't think it's a reasonable
>>>>> assumption
>>>>> >> that everybody will use python-openstackclient, is it?
>>>>> >>
>>>>> >> Said that, storing quotas in keystone poses a further challenge to
>>>>> the
>>>>> >> scalability of the system, which we shall perhaps address by using
>>>>> >> appropriate caching strategies and leveraging keystone
>>>>> notifications. Until
>>>>> >> we get that, I think that the openstack client will be the best way
>>>>> of
>>>>> >> getting a unified quota management experience.
>>>>> >>
>>>>> >> Salvatore
>>>>> >>
>>>>> >>
>>>>> >>>>
>>>>> >>>> Cheers,
>>>>> >>>> Morgan
>>>>> >>>>
>>>>> >>>> Sent via mobile
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Friday, October 3, 2014, Salvatore Orlando <
>>>>> sorlando at nicira.com>
>>>>> >>>> wrote:
>>>>> >>>>>
>>>>> >>>>> Thanks Vish,
>>>>> >>>>>
>>>>> >>>>> this seems a very reasonable first step as well - and since most
>>>>> >>>>> projects would be enforcing quotas in the same way, the shared
>>>>> library would
>>>>> >>>>> be the logical next step.
>>>>> >>>>> After all this is quite the same thing we do with authZ.
>>>>> >>>>>
>>>>> >>>>> Duncan is expressing valid concerns which in my opinion can be
>>>>> addressed
>>>>> >>>>> with an appropriate design - and a decent implementation.
>>>>> >>>>>
>>>>> >>>>> Salvatore
>>>>> >>>>>
>>>>> >>>>> On 3 October 2014 18:25, Vishvananda Ishaya <
>>>>> vishvananda at gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>>>
>>>>> >>>>>> The proposal in the past was to keep quota enforcement local,
>>>>> but to
>>>>> >>>>>> put the resource limits into keystone. This seems like an
>>>>> obvious first
>>>>> >>>>>> step to me. Then a shared library for enforcing quotas with
>>>>> decent
>>>>> >>>>>> performance should be next. The quota calls in nova are
>>>>> extremely
>>>>> >>>>>> inefficient right now and it will only get worse when we try to
>>>>> add
>>>>> >>>>>> hierarchical projects and quotas.
>>>>> >>>>>>
>>>>> >>>>>> Vish
>>>>> >>>>>>
>>>>> >>>>>> On Oct 3, 2014, at 7:53 AM, Duncan Thomas <
>>>>> duncan.thomas at gmail.com>
>>>>> >>>>>> wrote:
>>>>> >>>>>>
>>>>> >>>>>>> Taking quota out of the service / adding remote calls for quota
>>>>> >>>>>>> management is going to make things fragile - you've somehow
>>>>> got to
>>>>> >>>>>>> deal with the cases where your quota manager is slow, goes
>>>>> away,
>>>>> >>>>>>> hiccups, drops connections etc. You'll also need some way of
>>>>> >>>>>>> reconciling actual usage against quota usage periodically, to
>>>>> detect
>>>>> >>>>>>> problems.
>>>>> >>>>>>>
>>>>> >>>>>>> On 3 October 2014 15:03, Salvatore Orlando <
>>>>> sorlando at nicira.com>
>>>>> >>>>>>> wrote:
>>>>> >>>>>>>> Hi,
>>>>> >>>>>>>>
>>>>> >>>>>>>> Quota management is currently one of those things where every
>>>>> >>>>>>>> openstack
>>>>> >>>>>>>> project does its own thing. While quotas are obviously
>>>>> managed in a
>>>>> >>>>>>>> similar
>>>>> >>>>>>>> way for each project, there are subtle differences which
>>>>> ultimately
>>>>> >>>>>>>> result
>>>>> >>>>>>>> in lack of usability.
>>>>> >>>>>>>>
>>>>> >>>>>>>> I recall that in the past there have been several calls for
>>>>> unifying
>>>>> >>>>>>>> quota
>>>>> >>>>>>>> management. The blueprint [1] for instance, hints at the
>>>>> possibility
>>>>> >>>>>>>> of
>>>>> >>>>>>>> storing quotas in keystone.
>>>>> >>>>>>>> On the other hand, the blazar project [2, 3] seems to aim at
>>>>> solving
>>>>> >>>>>>>> this
>>>>> >>>>>>>> problem for good enabling resource reservation and therefore
>>>>> >>>>>>>> potentially
>>>>> >>>>>>>> freeing openstack projects from managing and enforcing quotas.
>>>>> >>>>>>>>
>>>>> >>>>>>>> While Blazar is definetely a good thing to have, I'm not
>>>>> entirely
>>>>> >>>>>>>> sure we
>>>>> >>>>>>>> want to make it a "required" component for every deployment.
>>>>> Perhaps
>>>>> >>>>>>>> single
>>>>> >>>>>>>> projects should still be able to enforce quota. On the other
>>>>> hand,
>>>>> >>>>>>>> at least
>>>>> >>>>>>>> on paper, the idea of making Keystone "THE" endpoint for
>>>>> managing
>>>>> >>>>>>>> quotas,
>>>>> >>>>>>>> and then letting the various project enforce them, sounds
>>>>> promising
>>>>> >>>>>>>> - is
>>>>> >>>>>>>> there any reason for which this blueprint is stalled to the
>>>>> point
>>>>> >>>>>>>> that it
>>>>> >>>>>>>> seems forgotten now?
>>>>> >>>>>>>>
>>>>> >>>>>>>> I'm coming to the mailing list with these random questions
>>>>> about
>>>>> >>>>>>>> quota
>>>>> >>>>>>>> management, for two reasons:
>>>>> >>>>>>>> 1) despite developing and using openstack on a daily basis
>>>>> I'm still
>>>>> >>>>>>>> confused by quotas
>>>>> >>>>>>>> 2) I've found a race condition in neutron quotas and the fix
>>>>> is not
>>>>> >>>>>>>> trivial.
>>>>> >>>>>>>> So, rather than start coding right away, it might probably
>>>>> make more
>>>>> >>>>>>>> sense
>>>>> >>>>>>>> to ask the community if there is already a known better
>>>>> approach to
>>>>> >>>>>>>> quota
>>>>> >>>>>>>> management - and obviously enforcement.
>>>>> >>>>>>>>
>>>>> >>>>>>>> Thanks in advance,
>>>>> >>>>>>>> Salvatore
>>>>> >>>>>>>>
>>>>> >>>>>>>> [1]
>>>>> https://blueprints.launchpad.net/keystone/+spec/service-metadata
>>>>> >>>>>>>> [2] https://wiki.openstack.org/wiki/Blazar
>>>>> >>>>>>>> [3]
>>>>> https://review.openstack.org/#/q/project:stackforge/blazar,n,z
>>>>> >>>>>>>>
>>>>> >>>>>>>> _______________________________________________
>>>>> >>>>>>>> OpenStack-dev mailing list
>>>>> >>>>>>>> OpenStack-dev at lists.openstack.org
>>>>> >>>>>>>>
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>> >>>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>> --
>>>>> >>>>>>> Duncan Thomas
>>>>> >>>>>>>
>>>>> >>>>>>> _______________________________________________
>>>>> >>>>>>> OpenStack-dev mailing list
>>>>> >>>>>>> OpenStack-dev at lists.openstack.org
>>>>> >>>>>>>
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> _______________________________________________
>>>>> >>>>>> OpenStack-dev mailing list
>>>>> >>>>>> OpenStack-dev at lists.openstack.org
>>>>> >>>>>>
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>> >>>>>>
>>>>> >>>>>
>>>>> >>>>
>>>>> >>>> _______________________________________________
>>>>> >>>> OpenStack-dev mailing list
>>>>> >>>> OpenStack-dev at lists.openstack.org
>>>>> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>> >>>>
>>>>> >>>
>>>>> >>>
>>>>> >>> _______________________________________________
>>>>> >>> OpenStack-dev mailing list
>>>>> >>> OpenStack-dev at lists.openstack.org
>>>>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >> _______________________________________________
>>>>> >> OpenStack-dev mailing list
>>>>> >> OpenStack-dev at lists.openstack.org
>>>>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>> >>
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Davanum Srinivas :: https://twitter.com/dims
>>>>> >
>>>>> > _______________________________________________
>>>>> > OpenStack-dev mailing list
>>>>> > OpenStack-dev at lists.openstack.org
>>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>>
>> --
>> Kind Regards
>> Valeriy Ponomaryov
>> www.mirantis.com
>> vponomaryov at mirantis.com
>>
>
>
>
> --
> Kind Regards
> Valeriy Ponomaryov
> www.mirantis.com
> vponomaryov at mirantis.com
>  _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20141015/58c98a4d/attachment.html>