<div dir="ltr"><font face="georgia, serif">Thanks everyone for being there at the Design Summit talk on Cross project Quotas.</font><div><font face="georgia, serif"><br></font></div><div><font face="georgia, serif">Points from the discussion are captured in this etherpad[1] for reference.</font></div><div><font face="georgia, serif"><br></font></div><div><font face="georgia, serif">-Vilobh</font></div><div><font face="georgia, serif">[1] <a href="https://etherpad.openstack.org/p/newton-quota-library">https://etherpad.openstack.org/p/newton-quota-library</a></font></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 26, 2016 at 9:33 AM, Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 04/25/2016 02:05 PM, Joshua Harlow wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Is the generation stuff going to be exposed outside of the AP?<br>
</blockquote>
<br>
No, wasn't planning on it.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm sort of hoping not(?), because a service (if one everyone wanted to<br>
create it) for say zookeeper (or etcd or consul...) could use its<br>
built-in generation equivalent (every znode has a version that u can use<br>
to do equivalent things).<br>
<br>
Thus it's like the gist I posted earlier @<br>
<br>
<a href="https://gist.github.com/harlowja/e7175c2d76e020a82ae94467a1441d85" rel="noreferrer" target="_blank">https://gist.github.com/harlowja/e7175c2d76e020a82ae94467a1441d85</a><br>
<br>
So might be nice to not expose such a thing outside of the db-layer.<br>
<br>
Amrith Kumar wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Sat, 2016-04-23 at 21:41 +0000, Amrith Kumar wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Ok to beer and high bandwidth. FYI Jay the distributed high perf db we<br>
did a couple of years ago is now open source. Just saying. Mysql plug<br>
compatible ....<br>
</blockquote>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
-amrith<br>
<br>
<br>
--<br>
Amrith Kumar<br>
<a href="mailto:amrith@tesora.com" target="_blank">amrith@tesora.com</a><br>
<br>
<br>
-------- Original message --------<br>
From: Jay Pipes<<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>><br>
Date: 04/23/2016 4:10 PM (GMT-05:00)<br>
To: Amrith Kumar<<a href="mailto:amrith@tesora.com" target="_blank">amrith@tesora.com</a>>,<br>
<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a><br>
Cc: <a href="mailto:vilobhmm@yahoo-inc.com" target="_blank">vilobhmm@yahoo-inc.com</a>, <a href="mailto:nik.komawar@gmail.com" target="_blank">nik.komawar@gmail.com</a>, Ed Leafe<br>
<<a href="mailto:ed@leafe.com" target="_blank">ed@leafe.com</a>><br>
Subject: Re: [openstack-dev] More on the topic of DELIMITER, the Quota<br>
Management Library proposal<br>
<br>
<br>
Looking forward to arriving in Austin so that I can buy you a beer,<br>
Amrith, and have a high-bandwidth conversation about how you're<br>
wrong. :P<br>
</blockquote>
<br>
<br>
Jay and I chatted and it took a long time to come to an agreement<br>
because we weren't able to find any beer.<br>
<br>
Here's what I think we've agreed about. The library will store data in<br>
two tables;<br>
<br>
1. the detail table which stores the individual claims and the resource<br>
class.<br>
2. a generation table which stores the resource class and a generation.<br>
<br>
When a claim is received, the requestor performs the following<br>
operations.<br>
<br>
begin<br>
<br>
select sum(detail.claims) as total_claims,<br>
generation.resource as resource,<br>
generation.generation last_generation<br>
from detail, generation<br>
where detail.resource = generation.resource<br>
and generation.resource =<chosen resource; memory, cpu, ...><br>
group by generation.generation, generation.resource<br>
<br>
if total_claims + this_claim< limit<br>
insert into detail values (this_claim, resource)<br>
<br>
update generation<br>
set generation = generation + 1<br>
where generation = last_generation<br>
<br>
if @@rowcount = 1<br>
-- all good<br>
commit<br>
else<br>
rollback<br>
-- try again<br>
<br>
<br>
<br>
There will be some bootstrapping that will be required for the situation<br>
where there are no detail records for a given resource and so on but I<br>
think we can figure that out easily. The easiest way I can think of<br>
doing that is to lose the join and do both the queries (one against the<br>
detail and one against the generation table within the same<br>
transaction).<br>
<br>
Using the generation table update as the locking mechanism that prevents<br>
multiple requestors from making concurrent claims.<br>
<br>
So long as people don't go and try and read these tables and change<br>
tables outside of the methods that the library provides, we can<br>
guarantee that this is al safe and will not oversubscribe.<br>
<br>
-amrith<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Comments inline.<br>
<br>
On 04/23/2016 11:25 AM, Amrith Kumar wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Sat, 2016-04-23 at 10:26 -0400, Andrew Laski wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Fri, Apr 22, 2016, at 09:57 PM, Tim Bell wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I have reservations on f and g.<br>
<br>
<br>
On f., We have had a number of discussions in the past about<br>
centralising quota (e.g. Boson) and the project teams of the other<br>
components wanted to keep the quota contents ‘close’. This can<br>
always be reviewed further with them but I would hope for at least<br>
</blockquote></blockquote></blockquote>
a<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
standard schema structure of tables in each project for the<br>
</blockquote></blockquote></blockquote>
handling<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
of quota.<br>
<br>
<br>
On g., aren’t all projects now nested projects ? If we have the<br>
complexity of handling nested projects sorted out in the common<br>
library, is there a reason why a project would not want to support<br>
nested projects ?<br>
<br>
<br>
One other issue is how to do reconcilliation, each project needs<br>
</blockquote></blockquote></blockquote>
to<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
have a mechanism to re-calculate the current allocations and<br>
reconcile that with the quota usage. While in an ideal world, this<br>
should not be necessary, it would be for the foreseeable future,<br>
especially with a new implementation.<br>
<br>
</blockquote>
One of the big reasons that Jay and I have been pushing to remove<br>
reservations and tracking of quota in a separate place than the<br>
resources are actually used, e.g., an instance record in the Nova<br>
</blockquote></blockquote>
db,<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
is so that reconciliation is not necessary. For example, if RAM<br>
</blockquote></blockquote>
quota<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
usage is simply tracked as sum(instances.memory_mb) then you can be<br>
sure that usage is always up to date.<br>
</blockquote>
Uh oh, there be gremlins here ...<br>
<br>
I am positive that this will NOT work, see earlier conversations<br>
</blockquote>
about<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
isolation levels, and Jay's alternate solution.<br>
<br>
The way (I understand the issue, and Jay's solution) you get around<br>
</blockquote>
the<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
isolation levels trap is to NOT do your quota determinations based<br>
</blockquote>
on a<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
SUM(column) but rather based on the rowcount on a well crafted<br>
</blockquote>
UPDATE of<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
a single table that stored total quota.<br>
</blockquote>
No, we would do our quota calculations by doing a SUM(used) against<br>
the<br>
allocations table. There is no separate table that stored the total<br>
quota (or quota usage records). That's the source of the problem with<br>
the existing quota handling code in Nova. The generation field value<br>
is<br>
used to provide the consistent view of the actual resource usage<br>
records<br>
so that the INSERT operations for all claimed resources can be done in<br>
a<br>
transactional manner and will be rolled back if any other writer<br>
changes<br>
the amount of consumed resources on a provider (which of course would<br>
affect the quota check calculations).<br>
<br>
> You could also store a detail<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
claim record for each claim in an independent table that is<br>
</blockquote>
maintained<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
in the same database transaction if you so desire, that is optional.<br>
</blockquote>
The allocations table is the "detail claim record" table that you<br>
refer<br>
to above.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
My view of how this would work (which I described earlier as<br>
</blockquote>
building on<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Jay's solution) is that the claim flow would look like this:<br>
<br>
select total_used, generation<br>
from quota_claimed<br>
where tenant =<tenant> and resource = 'memory'<br>
</blockquote>
There is no need to keep a total_used value for anything. That is<br>
denormalized calculated data that merely adds a point of race<br>
contention. The quota check is against the *detail* table<br>
(allocations),<br>
which stores the *actual resource usage records*.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
begin transaction<br>
<br>
update quota_claimed<br>
set total_used = total_used + claim, generation =<br>
generation + 1<br>
where tenant =<tenant> and resource = 'memory'<br>
and generation = generation<br>
and total_used + claim< limit<br>
</blockquote>
This part of the transaction must always occur **after** the<br>
insertion<br>
of the actual resource records, not before.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
if @@rowcount = 1<br>
-- optional claim_detail table<br>
insert into claim_detail values (<tenant>,<br>
</blockquote>
'memory',<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
claim, ...)<br>
commit<br>
else<br>
rollback<br>
</blockquote>
So, in pseudo-Python-SQLish code, my solution works like this:<br>
<br>
limits = get_limits_from_delimiter()<br>
requested = get_requested_from_request_spec()<br>
<br>
while True:<br>
<br>
used := SELECT<br>
resource_class,<br>
resource_provider,<br>
generation,<br>
SUM(used) as total_used<br>
FROM allocations<br>
JOIN resource_providers ON (...)<br>
WHERE consumer_uuid = $USER_UUID<br>
GROUP BY<br>
resource_class,<br>
resource_provider,<br>
generation;<br>
<br>
# Check that our requested resource amounts don't exceed quotas<br>
if not check_requested_within_limits(requested, used, limits):<br>
raise QuotaExceeded<br>
<br>
# Claim all requested resources. Note that the generation<br>
retrieved<br>
# from the above query is our consistent view marker. If the<br>
UPDATE<br>
# below succeeds and returns != 0 rows affected, that means there<br>
# was no other writer that changed our resource usage in between<br>
# this thread's claiming of resources, and therefore we prevent<br>
# any oversubscription of resources.<br>
begin_transaction:<br>
<br>
provider := SELECT id, generation, ... FROM<br>
resource_providers<br>
JOIN (...)<br>
WHERE (<resource_usage_filters>)<br>
<br>
for resource in requested:<br>
INSERT INTO allocations (<br>
resource_provider_id,<br>
resource_class_id,<br>
consumer_uuid,<br>
used<br>
) VALUES (<br>
$<a href="http://provider.id" rel="noreferrer" target="_blank">provider.id</a>,<br>
$<a href="http://resource.id" rel="noreferrer" target="_blank">resource.id</a>,<br>
$USER_UUID,<br>
$resource.amount<br>
);<br>
<br>
rows_affected := UPDATE resource_providers<br>
SET generation = generation + 1<br>
WHERE id = $<a href="http://provider.id" rel="noreferrer" target="_blank">provider.id</a><br>
AND generation =<br>
$used[$<a href="http://provider.id" rel="noreferrer" target="_blank">provider.id</a>].generation;<br>
<br>
if $rows_affected == 0:<br>
ROLLBACK;<br>
<br>
The only reason we would need a post-claim quota check is if some of<br>
the<br>
requested resources are owned and tracked by an external-to-Nova<br>
system.<br>
<br>
BTW, note to Ed Leafe... unless your distributed data store supports<br>
transactional semantics, you can't use a distributed data store for<br>
these types of solutions. Instead, you will need to write a whole<br>
bunch<br>
of code that does post-auditing of claims and quotas and a system<br>
that<br>
accepts that oversubscription and out-of-sync quota limits and usages<br>
is<br>
a fact of life. Not to mention needing to implement JOINs in Python.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
But, it is my understanding that<br>
<br>
(a) if you wish to do the SUM(column) approach that you<br>
</blockquote>
propose,<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
you must have a reservation that is committed and then you<br>
</blockquote>
must<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
re-read the SUM(column) to make sure you did not<br>
</blockquote>
over-subscribe;<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
and<br>
</blockquote>
Erm, kind of? Oversubscription is not possible in the solution I<br>
describe because the compare-and-update on the<br>
resource_providers.generation field allows for a consistent view of<br>
the<br>
resources used -- and if that view changes during the insertion of<br>
resource usage records -- the transaction containing those insertions<br>
is<br>
rolled back.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
(b) to get away from reservations you must stop using the<br>
SUM(column) approach and instead use a single quota_claimed<br>
table to determine the current quota claimed.<br>
</blockquote>
No. This has nothing to do with reservations.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
At least that's what I understand of Jay's example from earlier in<br>
</blockquote>
this<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
thread.<br>
<br>
Let's definitely discuss this in Austin. While I don't love Jay's<br>
solution for other reasons to do with making the quota table a<br>
</blockquote>
hotspot<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
and things like that, it is a perfectly workable solution, I think.<br>
</blockquote>
There is no quota table in my solution.<br>
<br>
If you refer to the resource_providers table (the table that has the<br>
generation field), then yes, it's a hot spot. But hot spots in the DB<br>
aren't necessarily a bad thing if you design the underlying schema<br>
properly.<br>
<br>
More in Austin.<br>
<br>
Best,<br>
-jay<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Tim<br>
<br>
<br>
<br>
<br>
From: Amrith Kumar<<a href="mailto:amrith@tesora.com" target="_blank">amrith@tesora.com</a>><br>
Reply-To: "OpenStack Development Mailing List (not for usage<br>
questions)"<<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
Date: Friday 22 April 2016 at 06:51<br>
To: "OpenStack Development Mailing List (not for usage questions)"<br>
<<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
Subject: Re: [openstack-dev] More on the topic of DELIMITER, the<br>
Quota Management Library proposal<br>
<br>
<br>
<br>
I’ve thought more about Jay’s approach to enforcing<br>
</blockquote></blockquote></blockquote>
quotas<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
and I think we can build on and around it. With that<br>
implementation as the basic quota primitive, I think we<br>
</blockquote></blockquote></blockquote>
can<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
build a quota management API that isn’t dependent on<br>
reservations. It does place some burdens on the consuming<br>
projects that I had hoped to avoid and these will cause<br>
heartburn for some (make sure that you always request<br>
resources in a consistent order and free them in a<br>
consistent order being the most obvious).<br>
<br>
<br>
<br>
<br>
<br>
If it doesn’t make it harder, I would like to see if we<br>
</blockquote></blockquote></blockquote>
can<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
make the quota API take care of the ordering of requests.<br>
i.e. if the quota API is an extension of Jay’s example<br>
</blockquote></blockquote></blockquote>
and<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
accepts some data structure (dict?) with all the claims<br>
</blockquote></blockquote></blockquote>
that<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
a project wants to make for some operation, and then<br>
proceeds to make those claims for the project in the<br>
consistent order, I think it would be of some value.<br>
<br>
<br>
<br>
<br>
<br>
Beyond that, I’m on board with a-g below,<br>
<br>
<br>
<br>
<br>
<br>
-amrith<br>
<br>
<br>
<br>
<br>
<br>
From: Vilobh Meshram<br>
[mailto:<a href="mailto:vilobhmeshram.openstack@gmail.com" target="_blank">vilobhmeshram.openstack@gmail.com</a>]<br>
Sent: Friday, April 22, 2016 4:08 AM<br>
To: OpenStack Development Mailing List (not for usage<br>
questions)<<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
Subject: Re: [openstack-dev] More on the topic of<br>
</blockquote></blockquote></blockquote>
DELIMITER,<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
the Quota Management Library proposal<br>
<br>
<br>
<br>
<br>
<br>
I strongly agree with Jay on the points related to "no<br>
reservation" , keeping the interface simple and the role<br>
</blockquote></blockquote></blockquote>
for<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Delimiter (impose limits on resource consumption and<br>
</blockquote></blockquote></blockquote>
enforce<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
quotas).<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
The point to keep user quota, tenant quotas in Keystone<br>
sounds interestring and would need support from Keystone<br>
team. We have a Cross project session planned [1] and<br>
</blockquote></blockquote></blockquote>
will<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
definitely bring that up in that session.<br>
<br>
<br>
<br>
<br>
<br>
<br>
The main thought with which Delimiter was formed was to<br>
enforce resource quota in transaction safe manner and do<br>
</blockquote></blockquote></blockquote>
it<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
in a cross-project conducive manner and it still holds<br>
true. Delimiters mission is to impose limits on<br>
resource consumption and enforce quotas in transaction<br>
</blockquote></blockquote></blockquote>
safe<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
manner. Few key aspects of Delimiter are :-<br>
<br>
<br>
<br>
a. Delimiter will be a new Library and not a Service.<br>
Details covered in spec.<br>
<br>
<br>
b. Delimiter's role will be to impose limits on resource<br>
consumption.<br>
<br>
<br>
c. Delimiter will not be responsible for rate limiting.<br>
<br>
<br>
d. Delimiter will not maintain data for the resources.<br>
Respective projects will take care of keeping,<br>
</blockquote></blockquote></blockquote>
maintaining<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
data for the resources and resource consumption.<br>
<br>
<br>
e. Delimiter will not have the concept of "reservations".<br>
Delimiter will read or update the "actual" resource<br>
</blockquote></blockquote></blockquote>
tables<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
and will not rely on the "cached" tables. At present, the<br>
quota infrastructure in Nova, Cinder and other projects<br>
</blockquote></blockquote></blockquote>
have<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
tables such as reservations, quota_usage, etc which are<br>
</blockquote></blockquote></blockquote>
used<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
as "cached tables" to track re<br>
<br>
<br>
f. Delimiter will fetch the information for project<br>
</blockquote></blockquote></blockquote>
quota,<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
user quota from a centralized place, say Keystone, or if<br>
that doesn't materialize will fetch default quota values<br>
from respective service. This information will be cached<br>
since it gets updated rarely but read many times.<br>
<br>
<br>
g. Delimiter will take into consideration whether the<br>
project is a Flat or Nested and will make the<br>
</blockquote></blockquote></blockquote>
calculations<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
of allocated, available resources. Nested means project<br>
namespace is hierarchical and Flat means project<br>
</blockquote></blockquote></blockquote>
namespace<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
is not hierarchical.<br>
<br>
<br>
-Vilobh<br>
<br>
<br>
[1]<br>
</blockquote></blockquote></blockquote>
<a href="https://www.openstack.org/summit/austin-2016/summit-schedule/events/9492" rel="noreferrer" target="_blank">https://www.openstack.org/summit/austin-2016/summit-schedule/events/9492</a><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
<br>
<br>
<br>
On Thu, Apr 21, 2016 at 11:08 PM, Joshua Harlow<br>
<<a href="mailto:harlowja@fastmail.com" target="_blank">harlowja@fastmail.com</a>> wrote:<br>
<br>
<br>
Since people will be on a plane soon,<br>
<br>
I threw this together as a example of a quota<br>
</blockquote></blockquote></blockquote>
engine<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
(the zookeeper code does even work, and yes it<br>
provides transactional semantics due to the nice<br>
abilities of zookeeper znode versions[1] and its<br>
inherent consistency model, yippe).<br>
<br>
<br>
</blockquote></blockquote></blockquote>
<a href="https://gist.github.com/harlowja/e7175c2d76e020a82ae94467a1441d85" rel="noreferrer" target="_blank">https://gist.github.com/harlowja/e7175c2d76e020a82ae94467a1441d85</a><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Someone else can fill in the db quota engine with<br>
</blockquote></blockquote></blockquote>
a<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
similar/equivalent api if they so dare, ha. Or<br>
</blockquote></blockquote></blockquote>
even<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
feel to say the gist/api above is crap, cause<br>
</blockquote></blockquote></blockquote>
that's<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
ok to, lol.<br>
<br>
[1]<br>
<br>
</blockquote></blockquote></blockquote>
<a href="https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#Data" rel="noreferrer" target="_blank">https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#Data</a><br>
+Access<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
Amrith Kumar wrote:<br>
<br>
Inline below ... thread is too long, will<br>
catch you in Austin.<br>
<br>
<br>
-----Original Message-----<br>
From: Jay Pipes<br>
[mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>]<br>
Sent: Thursday, April 21, 2016<br>
</blockquote></blockquote></blockquote>
8:08<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
PM<br>
To:<br>
<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a><br>
Subject: Re: [openstack-dev] More<br>
</blockquote></blockquote></blockquote>
on<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
the topic of DELIMITER, the Quota<br>
Management Library proposal<br>
<br>
Hmm, where do I start... I think<br>
</blockquote></blockquote></blockquote>
I<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
will just cut to the two primary<br>
disagreements I have. And I will<br>
top-post because this email is<br>
</blockquote></blockquote></blockquote>
way<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
too<br>
big.<br>
<br>
1) On serializable isolation<br>
</blockquote></blockquote></blockquote>
level.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
No, you don't need it at all to<br>
prevent races in claiming. Just<br>
</blockquote></blockquote></blockquote>
use<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
a<br>
compare-and-update with retries<br>
strategy. Proof is here:<br>
<br>
<br>
</blockquote></blockquote></blockquote>
<a href="https://github.com/jaypipes/placement-bench/blob/master/placement.py#L97-" rel="noreferrer" target="_blank">https://github.com/jaypipes/placement-bench/blob/master/placement.py#L97-</a><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
L142<br>
<br>
Works great and prevents multiple<br>
writers from oversubscribing any<br>
resource without relying on any<br>
particular isolation level at<br>
</blockquote></blockquote></blockquote>
all.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The `generation` field in the<br>
inventories table is what allows<br>
multiple<br>
writers to ensure a consistent<br>
</blockquote></blockquote></blockquote>
view<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
of the data without needing to<br>
</blockquote></blockquote></blockquote>
rely<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
on<br>
heavy lock-based semantics and/or<br>
RDBMS-specific isolation levels.<br>
<br>
<br>
<br>
[amrith] this works for what it is doing,<br>
</blockquote></blockquote></blockquote>
we<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
can definitely do this. This will work at<br>
any isolation level, yes. I didn't want<br>
</blockquote></blockquote></blockquote>
to<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
go this route because it is going to<br>
</blockquote></blockquote></blockquote>
still<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
require an insert into another table<br>
recording what the actual 'thing' is that<br>
</blockquote></blockquote></blockquote>
is<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
claiming the resource and that insert is<br>
going to be in a different transaction<br>
</blockquote></blockquote></blockquote>
and<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
managing those two transactions was what<br>
</blockquote></blockquote></blockquote>
I<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
wanted to avoid. I was hoping to avoid<br>
having two tables tracking claims, one<br>
showing the currently claimed quota and<br>
another holding the things that claimed<br>
</blockquote></blockquote></blockquote>
that<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
quota. Have to think again whether that<br>
</blockquote></blockquote></blockquote>
is<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
possible.<br>
<br>
2) On reservations.<br>
<br>
The reason I don't believe<br>
reservations are necessary to be<br>
</blockquote></blockquote></blockquote>
in<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
a quota<br>
library is because reservations<br>
</blockquote></blockquote></blockquote>
add<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
a concept of a time to a claim of<br>
some<br>
resource. You reserve some<br>
</blockquote></blockquote></blockquote>
resource<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
to be claimed at some point in<br>
</blockquote></blockquote></blockquote>
the<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
future and release those<br>
</blockquote></blockquote></blockquote>
resources<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
at a point further in time.<br>
<br>
Quota checking doesn't look at<br>
</blockquote></blockquote></blockquote>
what<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
the state of some system will be<br>
</blockquote></blockquote></blockquote>
at<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
some point in the future. It<br>
</blockquote></blockquote></blockquote>
simply<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
returns whether the system *right<br>
now* can handle a request *right<br>
now* to claim a set of resources.<br>
<br>
If you want reservation semantics<br>
for some resource, that's totally<br>
cool,<br>
but IMHO, a reservation service<br>
should live outside of the<br>
</blockquote></blockquote></blockquote>
service<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
that is<br>
actually responsible for<br>
</blockquote></blockquote></blockquote>
providing<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
resources to a consumer.<br>
Merging right-now quota checks<br>
</blockquote></blockquote></blockquote>
and<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
future-based reservations into<br>
</blockquote></blockquote></blockquote>
the<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
same<br>
library just complicates things<br>
unnecessarily IMHO.<br>
<br>
<br>
<br>
[amrith] extension of the above ...<br>
<br>
3) On resizes.<br>
<br>
Look, I recognize some users see<br>
some value in resizing their<br>
resources.<br>
That's fine. I personally think<br>
expand operations are fine, and<br>
</blockquote></blockquote></blockquote>
that<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
shrink operations are really the<br>
operations that should be<br>
</blockquote></blockquote></blockquote>
prohibited<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
in<br>
the API. But, whatever, I'm fine<br>
with resizing of requested<br>
</blockquote></blockquote></blockquote>
resource<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
amounts. My big point is if you<br>
don't have a separate table that<br>
stores<br>
quota_usages and instead only<br>
</blockquote></blockquote></blockquote>
have a<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
single table that stores the<br>
</blockquote></blockquote></blockquote>
actual<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
resource usage records, you don't<br>
have to do *any* quota check<br>
operations<br>
at all upon deletion of a<br>
</blockquote></blockquote></blockquote>
resource.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
For modifying resource amounts<br>
</blockquote></blockquote></blockquote>
(i.e.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
a<br>
resize) you merely need to change<br>
the calculation of requested<br>
resource<br>
amounts to account for the<br>
already-consumed usage amount.<br>
<br>
Bottom line for me: I really<br>
</blockquote></blockquote></blockquote>
won't<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
support any proposal for a<br>
</blockquote></blockquote></blockquote>
complex<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
library that takes the resource<br>
claim process out of the hands of<br>
the<br>
services that own those<br>
</blockquote></blockquote></blockquote>
resources.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The simpler the interface of this<br>
library, the better.<br>
<br>
<br>
<br>
[amrith] my proposal would not but this<br>
email thread has got too long. Yes,<br>
</blockquote></blockquote></blockquote>
simpler<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
interface, will catch you in Austin.<br>
<br>
Best,<br>
-jay<br>
<br>
On 04/19/2016 09:59 PM, Amrith<br>
</blockquote></blockquote></blockquote>
Kumar<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
wrote:<br>
<br>
-----Original<br>
Message-----<br>
From: Jay Pipes<br>
<br>
</blockquote></blockquote></blockquote>
[mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>]<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Sent: Monday,<br>
</blockquote></blockquote></blockquote>
April<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
18, 2016 2:54 PM<br>
To:<br>
<br>
</blockquote></blockquote></blockquote>
<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Subject: Re:<br>
[openstack-dev]<br>
</blockquote></blockquote></blockquote>
More<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
on the topic of<br>
DELIMITER, the<br>
Quota Management<br>
Library proposal<br>
<br>
On 04/16/2016<br>
</blockquote></blockquote></blockquote>
05:51<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
PM, Amrith Kumar<br>
wrote:<br>
<br>
If we<br>
therefore<br>
assume<br>
</blockquote></blockquote></blockquote>
that<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
this will<br>
</blockquote></blockquote></blockquote>
be<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
a Quota<br>
<br>
</blockquote></blockquote></blockquote>
Management<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Library,<br>
it is<br>
</blockquote></blockquote></blockquote>
safe<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
to assume<br>
that<br>
</blockquote></blockquote></blockquote>
quotas<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
are going<br>
</blockquote></blockquote></blockquote>
to<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
be<br>
</blockquote></blockquote></blockquote>
managed<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
on a<br>
<br>
</blockquote></blockquote></blockquote>
per-project<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
basis,<br>
</blockquote></blockquote></blockquote>
where<br>
participating projects will use this library.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I believe<br>
that it<br>
stands to<br>
reason<br>
</blockquote></blockquote></blockquote>
that<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
any data<br>
<br>
</blockquote></blockquote></blockquote>
persistence<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
will<br>
have to<br>
</blockquote></blockquote></blockquote>
be<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
in a<br>
location<br>
decided<br>
</blockquote></blockquote></blockquote>
by<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
the<br>
<br>
</blockquote></blockquote></blockquote>
individual<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
project.<br>
<br>
<br>
Depends on what<br>
</blockquote></blockquote></blockquote>
you<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
mean by "any data<br>
persistence". If<br>
</blockquote></blockquote></blockquote>
you<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
are<br>
referring to the<br>
storage of quota<br>
values (per user,<br>
per tenant,<br>
global, etc) I<br>
</blockquote></blockquote></blockquote>
think<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
that should be<br>
</blockquote></blockquote></blockquote>
done<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
by the Keystone<br>
service.<br>
This data is<br>
essentially an<br>
attribute of the<br>
user or the<br>
</blockquote></blockquote></blockquote>
tenant<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
or the<br>
<br>
<br>
service endpoint itself (i.e.<br>
<br>
<br>
global defaults).<br>
This data also<br>
rarely changes<br>
</blockquote></blockquote></blockquote>
and<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
logically belongs<br>
to the service<br>
</blockquote></blockquote></blockquote>
that<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
manages users,<br>
tenants, and<br>
</blockquote></blockquote></blockquote>
service<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
endpoints:<br>
<br>
<br>
Keystone.<br>
<br>
<br>
If you are<br>
</blockquote></blockquote></blockquote>
referring<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
to the storage of<br>
resource usage<br>
records, yes,<br>
each service<br>
</blockquote></blockquote></blockquote>
project<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
should own that<br>
</blockquote></blockquote></blockquote>
data<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
(and frankly, I<br>
don't see a<br>
need to persist<br>
</blockquote></blockquote></blockquote>
any<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
quota usage data<br>
</blockquote></blockquote></blockquote>
at<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
all, as I<br>
</blockquote></blockquote></blockquote>
mentioned<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
in a<br>
previous reply to<br>
Attila).<br>
<br>
<br>
[amrith] You make a<br>
distinction that I had<br>
</blockquote></blockquote></blockquote>
made<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
implicitly, and it is<br>
important to highlight<br>
</blockquote></blockquote></blockquote>
it.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Thanks for pointing it<br>
</blockquote></blockquote></blockquote>
out.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Yes, I meant</blockquote></blockquote></blockquote></blockquote></blockquote></blockquote>
</blockquote></div><br></div>