Open Stack

Hi,
Interesting  thread. I’ve just started looking at Boson myself to manage quotas across multiple regions. I think one of the cases when having quota management as a separate service is justified, is a multi-region case [1].

1) Some deployments can share Keystone, so there’s a need to synchronise resource usage across multiple OpenStacks and enforce quota in a distributed manner. Architecturally this will mean that there will be one Boson instance with the “quota management” and “admin” roles exposing REST API endpoint and talking REST to other client specific Boson instances. This use case can be an example of why it can be reasonable to have a separate service for managing quotas, reservations, usages.

2) As I understand it, the intention of Boson is to synchronise usage, not own this information. The “Usage synchronisation” paragraph in the wiki [3] describes one possible approach:
“…Boson will keep freshness information on the usage data; should it determine that the usage information it has is not fresh enough, it will reject reservation creation requests with a special response code, which tells the service to send fresh usage information for certain resources along with resending the reservation creation requests”.
Further, the service can reject reservation creation requests in case of communication failure, tracker unavailability, etc.

3) Have you looked at Blazar [3] as a possible reservation mechanism instead of implementing a new reservation interface in Boson? Does it at all make sense to use it in the context of quota management?

Regards,
Dimitri

[1] http://dc636.4shared.com/download/Z6O_jJSGba/multiregion_arch.png?lgfp=3000
[2]https://wiki.openstack.org/wiki/Boson#Usage_Synchronization
[3] https://wiki.openstack.org/wiki/Blazar

From: Salvatore Orlando <sorlando at nicira.com<mailto:sorlando at nicira.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Thursday 15 January 2015 02:22
To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Subject: Re: [openstack-dev] Quota management and enforcement across projects

I'm resurrecting this thread to provide an update on this effort.

I have been looking at Boson [1] as a base for starting developing a service for managing quotas and reservations [2].
Boson's model would satisfy most of the requirements for this service, and implementing additional requirements, such as hierarchical multi-tenancy should be quite easy (at least from the high-level design perspective). In the rest of this post I'm going to refer to this service both as "Boson" and "quota service".

Without going into too many technical details (which I am happy to discuss separately), the quota service will need to provide the 4 interfaces depicted in [3].
1) The admin interface has the main purpose of keeping track of the services which use boson, registering resources to track, and managing their lifecycle.
2) The "quota mgmt" interface does pretty much what the "quota extension" does for many openstack project - manage resource limits per project/users, configure quota classes, etc.
3) The "reservation" interface handles the request/commit/cancel process supported by many Openstack projects
4) the "usage" interface provides abstraction to access the resource usage tracker and feed information to it.

These interfaces are then implemented by the Boson object model, depicted in [4].
This object model is not very different from the one originally proposed for Boson [5]
The proposed object model simplifies a bit the original one, merging the "reservation" and "request" concepts, as well as doing without the "SpecificResource" concept (at least for the moment). However, the proposed object model adds the "Quota class" concept and introduces the possibility of having child-parent relationships between Resources and User Info. The former will allow for applying quota to resources which are scoped within a parent resource (e.g.: static routes per logical router), whereas the latter should enable the hierarchical multi-tenancy use case.

Keeping in mind the interfaces discussed in [3], the component diagram [6] can be devised. There is a distinct component for each interface - plus one component for DB management. Most of the interactions are, of course, with the DB manager. Component design should be done in a way that the various components are as independent as possible. There are some interactions among components, but they can likely be replaced with interactions with the "DB manager" component.

The Boson quota service therefore represents a centralized endpoint for managing quotas, tracking resource usage, and performing resource reservation.
Conceptually, this is all good; however, would it scale from an architectural perspective? The main problem with this approach indeed is that the quota service itself can become a huge bottleneck and add more latency to resource processing. This problem can probably be solved looking at the above mentioned interfaces as "roles", and expecting Boson to scale horizontally via deployment of several instances of the service which can implement different roles, some of which can be dedicated to specific services.

For instance, let's consider the example architecture depicted in [7]. Here there are 3 boson instances.
One instance has the "quota management" and "admin" role and takes care of registering resources for enabled services, and setting/retrieving resource limits. It might expose its own REST API endpoint over HTTP to this aim.
The other two instances have the "reservation" and "usage" roles, and communicate with client applications over AMQP. However one instance will listen only to "neutron" messages and the other only to "nova" messages (nova and neutron have been used merely as an example, the same applies to any project). This architecture will enable some sort of horizontal scale out as every project will then have its own instance for tracking resources and managing reservations.
However, there will still be need for some communication across the various Boson instances. For instance in order to make a reservation, current resource limits should be fetched interacting with the management component or from the database. This overhead can be reduced considerably by introducing caches, which would result particularly effective for data such as resource limits which do not tend to change a lot over time.
Finally, it is also worth noting that in theory nothing prevents us from running Boson's management interface as a Keystone 3rd party component and therefore running its API on the Keystone endpoint.

Summarising, I hope I managed to convey the essentials of where I think Boson should be place in the OpenStack ecosystem, and how would it work from a conceptual and architectural perspective. I am aware that this is not exhaustive at al, but is probably the best that can be achieved in a mailing list post.

Nevertheless, there are still some fundamental questions that I am trying to answer in a convincing way.

1) would anyone ever deploy a service aimed exclusively at managing quotas and reservations?
This clearly makes an OpenStack deployment even more complex. The additional complexity might outweigh the hassle of having quota management API sparse over multiple projects and reservations performed in an inconsistent way. Operators might an additional orchestration service for the former, where as the latter issue might be solved either ensuring projects do reservations in the same way, or through the library idea we rejected (at least in terms of oslo library).

2) the reliability of a system where resource usage tracking is delegated to an external project.
Clearly there will be many more chances of actual resource usage to differ from the one represented in the tracker. This might easily happen in case of communication failure, temporary unavailability of the tracker, failures during requests processing, etc. etc.

3) The scalability of a reservation process handled within the project versus a reservation process delegated to an external system.

I am currently adapting Boson's code to the new object model and "synchronizing" it with the current OpenStack world - adding for instance support for objects like 'nova objects'. However, I would like to seek your input both on the devised architecture and most importantly on the above questions - mostly because at the end of the day I am not entirely convinced a service like this would actually be adopted by operators.

Regards,
Salvatore

[1] https://github.com/klmitch/boson/
[2] If your thinking of a funny name for this service, stop. If your thinking that this is "quota as a service", stop.
[3] http://www.4shared.com/download/-NJIub0fce/boson_overview.jpg?lgfp=3000
[4] http://www.4shared.com/download/a38OINvPba/boson_obj_model.jpg?lgfp=3000
[5] https://github.com/klmitch/boson/blob/master/doc/data-model.png
[6] http://www.4shared.com/download/3CNdnsmuce/boson_components.jpg?lgfp=3000
[7] http://www.4shared.com/download/ds0VRvNlba/boson_arch.jpg?lgfp=3000

On 28 November 2014 at 18:20, Jay Pipes <jaypipes at gmail.com<mailto:jaypipes at gmail.com>> wrote:
On 11/17/2014 05:49 PM, Salvatore Orlando wrote:
Hi all,

I am resuming this thread following the session we had at the summit in
Paris (etherpad here [1])

While there was some sort of consensus regarding what this library
should do, and how it should do it, the session ended with some open
questions which we need to address before finalising the specification.

There was a rather large consensus that the only viable architecture
would be one where the quota management library owns resource usage
data. However, this raises further concerns around:
- ownership of resource usage records. As the quota library owns usage
data, it becomes the authoritative source of truth for it. This could be
problematic in some projects, particularly nova, where the resource
tracker currently own usage data.

Well, just to be clear... the resource tracker in Nova owns the usage records for the compute node, not the usage records for a tenant or user (the quota driver and DB tables own that data).

Best,
-jay


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150324/d9853006/attachment.html>