[Openstack-operators] max_age and until_refresh for fixing Nova quotas

Sam Morrison sorrison at gmail.com
Fri Mar 20 23:53:02 UTC 2015


I put the script up at https://gist.github.com/sorrison/99c8e87295756e0ed787

The cron job that runs is like:

#!/bin/bash
BASEDIR=/tmp
/usr/bin/mysql --defaults-group-suffix=-prod-keystone -B -e 'select target_id, actor_id from assignment inner join project on target_id = project.id inner join user on user.id = actor_id order by target_id, actor_id;' | tail -n+2 > $BASEDIR/keystone_project_users.txt
scp $BASEDIR/keystone_project_users.txt root at nova-controller:.
ssh root at nova-controller nova-manage shell script sync_quotas_main.py


It’s not pretty but it works…..

Sam



> On 21 Mar 2015, at 12:35 am, Kris G. Lindgren <klindgren at godaddy.com> wrote:
> 
> Can you post of you cronjob/script that you use to correct the quotas?
> ____________________________________________
> 
> Kris Lindgren
> Senior Linux Systems Engineer
> GoDaddy, LLC.
> 
> 
> 
> On 3/20/15, 4:18 AM, "Sam Morrison" <sorrison at gmail.com> wrote:
> 
>> We¹ve had the following for a year or so but doesn¹t help much, we still
>> see it occurring every 10 mins or so.
>> 
>> max_age = 10
>> until_refresh = 5
>> reservation_expire=600
>> 
>> We have a cron job that runs every 10 mins that figures out what projects
>> are out of sync and corrects them.
>> We¹ve always been scared of setting these to zero but we probably should.
>> 
>> Sam
>> 
>> 
>>> On 15 Mar 2015, at 2:53 pm, Mike Dorman <mdorman at godaddy.com> wrote:
>>> 
>>> Yeah the default is just Œ0¹ for both, which disables the refresh.
>>> 
>>> 
>>> 
>>> The one downside is that it may not be 100% transparent to the user.
>>> If 
>>> the quota is already (incorrectly) too high, and exceeding the quota
>>> limit, the reservation that triggers the refresh will still fail.  I.e.
>>> the reservation is attempted based on the quota usage values _before_
>>> the 
>>> refresh.  But then after that the quota should be fixed and it will
>>> work 
>>> again on the next reservation.
>>> 
>>> But my thinking is that most quota issues happen slowly over time.  If
>>> we 
>>> are correcting them often and automatically, they hopefully never get
>>> to 
>>> the point where they¹re bad enough to manifest reservation errors to
>>> the 
>>> user.
>>> 
>>> I don¹t have any information re: db load.  I assume it regenerates
>>> based 
>>> on what¹s in the instances or reservations table.  I imagine the load
>>> for 
>>> doing a single refresh is probably comparable to doing a Œnova list¹.
>>> 
>>> Mike
>>> 
>>> 
>>> 
>>> On 3/14/15, 2:27 PM, "Tim Bell" <Tim.Bell at cern.ch> wrote:
>>> 
>>>> Interesting... what are the defaults ?
>>>> 
>>>> Assuming no massive DB load, getting synced within a day would seem
>>>> reasonable. Is the default no max age ?
>>>> 
>>>> Tim
>>>> 
>>>>> -----Original Message-----
>>>>> From: Jesse Keating [mailto:jlk at bluebox.net]
>>>>> Sent: 14 March 2015 16:59
>>>>> To: openstack-operators at lists.openstack.org
>>>>> Subject: Re: [Openstack-operators] max_age and until_refresh for
>>>>> fixing 
>>>>> Nova
>>>>> quotas
>>>>> 
>>>>> On 3/14/15 8:11 AM, Mike Dorman wrote:
>>>>>> I did short write-up here http://t.co/Q5X1hTgJG1 if you are
>>>>>> interested
>>>>>> in the details.
>>>>>> 
>>>>> 
>>>>> Thanks for sharing Matt! That's an excellent write up.
>>>>> 
>>>>> --
>>>>> -jlk
>>>>> 
>>>>> _______________________________________________
>>>>> OpenStack-operators mailing list
>>>>> OpenStack-operators at lists.openstack.org
>>>>> 
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>> 
>>>> _______________________________________________
>>>> OpenStack-operators mailing list
>>>> OpenStack-operators at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>> _______________________________________________
>>> OpenStack-operators mailing list
>>> OpenStack-operators at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> 
>> 
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> 




More information about the OpenStack-operators mailing list