<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>Hi,</div>
<div>Interesting thread. I’ve just started looking at Boson myself to manage quotas across multiple regions. I think one of the cases when having quota management as a separate service is justified, is a multi-region case [1].</div>
<div><br>
</div>
<div>1) Some deployments can share Keystone, so there’s a need to synchronise resource usage across multiple OpenStacks and enforce quota in a distributed manner. Architecturally this will mean that there will be one Boson instance with the “quota management”
and “admin” roles exposing REST API endpoint and talking REST to other client specific Boson instances. This use case can be an example of why it can be reasonable to have a separate service for managing quotas, reservations, usages. </div>
<div><br>
</div>
<div>2) As I understand it, the intention of Boson is to synchronise usage, not own this information. The “Usage synchronisation” paragraph in the wiki [3] describes one possible approach:</div>
<div>“…<span style="color: rgb(51, 51, 51); font-family: 'Arial Unicode MS', Arial, sans-serif; line-height: 20px; widows: 1; background-color: rgb(255, 255, 255);">Boson will keep freshness information on the usage data; should it determine that the usage
information it has is not fresh enough, it will reject reservation creation requests with a special response code, which tells the service to send fresh usage information for certain resources along with resending the reservation creation requests</span>”.</div>
<div>Further, the service can reject reservation creation requests in case of communication failure, tracker unavailability, etc.</div>
<div><br>
</div>
<div>3) Have you looked at Blazar [3] as a possible reservation mechanism instead of implementing a new reservation interface in Boson? Does it at all make sense to use it in the context of quota management?</div>
<div><br>
</div>
<div>Regards,</div>
<div>Dimitri</div>
<div><br>
</div>
<div>[1] <a href="http://dc636.4shared.com/download/Z6O_jJSGba/multiregion_arch.png?lgfp=3000">http://dc636.4shared.com/download/Z6O_jJSGba/multiregion_arch.png?lgfp=3000</a></div>
<div>[2]<a href="https://wiki.openstack.org/wiki/Boson#Usage_Synchronization">https://wiki.openstack.org/wiki/Boson#Usage_Synchronization</a></div>
<div>[3] <a href="https://wiki.openstack.org/wiki/Blazar">https://wiki.openstack.org/wiki/Blazar</a></div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Salvatore Orlando <<a href="mailto:sorlando@nicira.com">sorlando@nicira.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Thursday 15 January 2015 02:22<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] Quota management and enforcement across projects<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">I'm resurrecting this thread to provide an update on this effort.
<div><br>
</div>
<div>I have been looking at Boson [1] as a base for starting developing a service for managing quotas and reservations [2].</div>
<div>Boson's model would satisfy most of the requirements for this service, and implementing additional requirements, such as hierarchical multi-tenancy should be quite easy (at least from the high-level design perspective). In the rest of this post I'm going
to refer to this service both as "Boson" and "quota service".</div>
<div><br>
</div>
<div>Without going into too many technical details (which I am happy to discuss separately), the quota service will need to provide the 4 interfaces depicted in [3].</div>
<div>1) The admin interface has the main purpose of keeping track of the services which use boson, registering resources to track, and managing their lifecycle.</div>
<div>2) The "quota mgmt" interface does pretty much what the "quota extension" does for many openstack project - manage resource limits per project/users, configure quota classes, etc.</div>
<div>3) The "reservation" interface handles the request/commit/cancel process supported by many Openstack projects<br>
</div>
<div>4) the "usage" interface provides abstraction to access the resource usage tracker and feed information to it.</div>
<div><br>
</div>
<div>These interfaces are then implemented by the Boson object model, depicted in [4].</div>
<div>This object model is not very different from the one originally proposed for Boson [5]</div>
<div>The proposed object model simplifies a bit the original one, merging the "reservation" and "request" concepts, as well as doing without the "SpecificResource" concept (at least for the moment). However, the proposed object model adds the "Quota class"
concept and introduces the possibility of having child-parent relationships between Resources and User Info. The former will allow for applying quota to resources which are scoped within a parent resource (e.g.: static routes per logical router), whereas the
latter should enable the hierarchical multi-tenancy use case.</div>
<div><br>
</div>
<div>Keeping in mind the interfaces discussed in [3], the component diagram [6] can be devised. There is a distinct component for each interface - plus one component for DB management. Most of the interactions are, of course, with the DB manager. Component
design should be done in a way that the various components are as independent as possible. There are some interactions among components, but they can likely be replaced with interactions with the "DB manager" component.</div>
<div><br>
</div>
<div>The Boson quota service therefore represents a centralized endpoint for managing quotas, tracking resource usage, and performing resource reservation.</div>
<div>Conceptually, this is all good; however, would it scale from an architectural perspective? The main problem with this approach indeed is that the quota service itself can become a huge bottleneck and add more latency to resource processing. This problem
can probably be solved looking at the above mentioned interfaces as "roles", and expecting Boson to scale horizontally via deployment of several instances of the service which can implement different roles, some of which can be dedicated to specific services.</div>
<div><br>
</div>
<div>For instance, let's consider the example architecture depicted in [7]. Here there are 3 boson instances.</div>
<div>One instance has the "quota management" and "admin" role and takes care of registering resources for enabled services, and setting/retrieving resource limits. It might expose its own REST API endpoint over HTTP to this aim.</div>
<div>The other two instances have the "reservation" and "usage" roles, and communicate with client applications over AMQP. However one instance will listen only to "neutron" messages and the other only to "nova" messages (nova and neutron have been used merely
as an example, the same applies to any project). This architecture will enable some sort of horizontal scale out as every project will then have its own instance for tracking resources and managing reservations.</div>
<div>However, there will still be need for some communication across the various Boson instances. For instance in order to make a reservation, current resource limits should be fetched interacting with the management component or from the database. This overhead
can be reduced considerably by introducing caches, which would result particularly effective for data such as resource limits which do not tend to change a lot over time.</div>
<div>Finally, it is also worth noting that in theory nothing prevents us from running Boson's management interface as a Keystone 3rd party component and therefore running its API on the Keystone endpoint.<br>
</div>
<div><br>
</div>
<div>Summarising, I hope I managed to convey the essentials of where I think Boson should be place in the OpenStack ecosystem, and how would it work from a conceptual and architectural perspective. I am aware that this is not exhaustive at al, but is probably
the best that can be achieved in a mailing list post.</div>
<div><br>
</div>
<div>Nevertheless, there are still some fundamental questions that I am trying to answer in a convincing way.</div>
<div><br>
</div>
<div>1) would anyone ever deploy a service aimed exclusively at managing quotas and reservations?</div>
<div>This clearly makes an OpenStack deployment even more complex. The additional complexity might outweigh the hassle of having quota management API sparse over multiple projects and reservations performed in an inconsistent way. Operators might an additional
orchestration service for the former, where as the latter issue might be solved either ensuring projects do reservations in the same way, or through the library idea we rejected (at least in terms of oslo library).</div>
<div><br>
</div>
<div>2) the reliability of a system where resource usage tracking is delegated to an external project.</div>
<div>Clearly there will be many more chances of actual resource usage to differ from the one represented in the tracker. This might easily happen in case of communication failure, temporary unavailability of the tracker, failures during requests processing,
etc. etc. </div>
<div><br>
</div>
<div>3) The scalability of a reservation process handled within the project versus a reservation process delegated to an external system.</div>
<div><br>
</div>
<div>I am currently adapting Boson's code to the new object model and "synchronizing" it with the current OpenStack world - adding for instance support for objects like 'nova objects'. However, I would like to seek your input both on the devised architecture
and most importantly on the above questions - mostly because at the end of the day I am not entirely convinced a service like this would actually be adopted by operators.</div>
<div><br>
</div>
<div>Regards,</div>
<div>Salvatore</div>
<div><br>
</div>
<div>[1] <a href="https://github.com/klmitch/boson/">https://github.com/klmitch/boson/</a><br>
</div>
<div>[2] If your thinking of a funny name for this service, stop. If your thinking that this is "quota as a service", stop.</div>
<div>[3] <a href="http://www.4shared.com/download/-NJIub0fce/boson_overview.jpg?lgfp=3000">http://www.4shared.com/download/-NJIub0fce/boson_overview.jpg?lgfp=3000</a></div>
<div>[4] <a href="http://www.4shared.com/download/a38OINvPba/boson_obj_model.jpg?lgfp=3000">
http://www.4shared.com/download/a38OINvPba/boson_obj_model.jpg?lgfp=3000</a></div>
<div>[5] <a href="https://github.com/klmitch/boson/blob/master/doc/data-model.png">https://github.com/klmitch/boson/blob/master/doc/data-model.png</a></div>
<div>[6] <a href="http://www.4shared.com/download/3CNdnsmuce/boson_components.jpg?lgfp=3000">http://www.4shared.com/download/3CNdnsmuce/boson_components.jpg?lgfp=3000</a></div>
<div>[7] <a href="http://www.4shared.com/download/ds0VRvNlba/boson_arch.jpg?lgfp=3000">http://www.4shared.com/download/ds0VRvNlba/boson_arch.jpg?lgfp=3000</a></div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 28 November 2014 at 18:20, Jay Pipes <span dir="ltr">
<<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span class="">On 11/17/2014 05:49 PM, Salvatore Orlando wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi all,<br>
<br>
I am resuming this thread following the session we had at the summit in<br>
Paris (etherpad here [1])<br>
<br>
While there was some sort of consensus regarding what this library<br>
should do, and how it should do it, the session ended with some open<br>
questions which we need to address before finalising the specification.<br>
<br>
There was a rather large consensus that the only viable architecture<br>
would be one where the quota management library owns resource usage<br>
data. However, this raises further concerns around:<br>
- ownership of resource usage records. As the quota library owns usage<br>
data, it becomes the authoritative source of truth for it. This could be<br>
problematic in some projects, particularly nova, where the resource<br>
tracker currently own usage data.<br>
</blockquote>
<br>
</span>Well, just to be clear... the resource tracker in Nova owns the usage records for the compute node, not the usage records for a tenant or user (the quota driver and DB tables own that data).<br>
<br>
Best,<br>
-jay
<div class="HOEnZb">
<div class="h5"><br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</span>
</body>
</html>