[Openstack] Quotas... 1 of 1 instances? What's the deal?

Kevin L. Mitchell kevin.mitchell at rackspace.com
Tue Jun 5 17:31:22 UTC 2012


On Tue, 2012-06-05 at 13:03 -0400, Jay Pipes wrote:
> On 06/04/2012 05:43 PM, Kevin L. Mitchell wrote:
> > One thing to check is the total quotas on memory and disk, relative to
> > the size of the instance.  The original code computes a maximum number
> > of instances based on those values; my new code simply tries to emulate
> > that computation.  (I'm pretty sure I got it right, but honestly quotas
> > needs further rototillings…)
> 
> Total is 24G and 12 cores available on box, so no probs there...

Available on the box, or available in your quotas?

> > Hmmm…when quiesced, you should only see reservations if instances are
> > actively building.  (In fact, reservations should be committed as soon
> > as the instances are created in the database.)  It's possible that I
> > missed places where instances are created, but I thought I got them all…
> 
> OK, so I checked again this morning and I'm still seeing the project 
> beginning with 287 above (the demo tenant) having 10 reserved, no 
> in_use, and a NULL until_refresh column. When is this quiescience point 
> supposed to come?

As I say, what's supposed to happen is that quotas are checked and
reservations created in one transaction; then you create the appropriate
database entries, and once that's done, you commit the reservations, at
which point "reserved" should revert to zero.  That this hasn't happened
is a clear indication of a bug.

Reservations expire, by default, after 24 hours (86400 seconds), and
that should also cause "reserved" to go to zero.  Check your
reservations table and see if the reservations tally up correctly, and
see if they've expired?  They should be expired by a periodic task on
the scheduler…

> When I check the instances table for that project_id, look what pops up:
> 
> mysql> select vm_state, task_state, count(*) as num from instances where 
> project_id = '287a92da0cf14a27a43c8737417b029d' group by vm_state, 
> task_state;
> +----------+------------+-----+
> | vm_state | task_state | num |
> +----------+------------+-----+
> | building | scheduling |   5 |
> | deleted  | NULL       |  68 |
> +----------+------------+-----+
> 2 rows in set (0.00 sec)

So, only 5 instances, but they don't show up in "nova list"?  *boggle*

> I'm restarting my devstack + tempest runs with SCREEN_LOGDIR set so I 
> can track what's happening in the scheduler to leave the instances in 
> scheduling state. But clearly, something is haywire with either:
> 
>   * The error handling in how quotas are calculated (since by the DB, 
> there are only 5 out of the 10 quota-limited instances known to be 
> "scheduling"
>   * The reservation system is not freeing up resources after some amount 
> of time? Is there a configuration option for setting the expiration time 
> for reservations?

The expiration time on reservations is set using the reservation_expire
configuration option.  As mentioned, it defaults to 86400 seconds, and
the scheduler's _expire_reservations() periodic task is supposed to
trigger the reservation expiration.

(And I'm all for finding out what I missed and getting it fixed!)
-- 
Kevin L. Mitchell <kevin.mitchell at rackspace.com>





More information about the Openstack mailing list