Open Stack

I'm floating this in case the idea has come up before.

I've spent most of my day so far digging through logs and code for gate 
race bug 1353962 [1] where an admin tenant is going over-quota when 
trying to allocate fixed IPs for a test instance (the test setup creates 
2 instances and more test instances are created in some of the test cases).

I'm going to push up some patches to nova to make the logs more useful 
when we hit this, i.e. log the quota limit and current usage for the 
given resource (fixed_ips in this case).

But the idea that keeps coming up when I'm digging into stuff like this 
is it'd be nice to know when we're nearing over-quota so we can walk 
that back and figure out what was going on prior to us falling over the 
cliff. This isn't a novel idea, but I haven't heard it discussed before.

What I'm thinking is basically add a config option for setting a 
threshold value which would dump warnings when you're over that 
threshold.  This could be configured per resource, e.g. I only care 
about fixed_ips, instances, etc.  It could be configured per tenant 
(maybe we only care about this for the admin tenant?).

My use case would be specifically related to the bug in question, so I 
want to set the threshold at let's say 80%.  With a default fixed_ips 
quota of 10, when I've gotten over 8 reserved for the admin tenant, I 
start logging warnings that I'm nearing over-quota.  When we go 
over-quota on a request, we could then see what operations/requests were 
going on before this that tipped us over.

Thoughts?

[1] https://bugs.launchpad.net/nova/+bug/1353962

-- 

Thanks,

Matt Riedemann