[Openstack-operators] max_age and until_refresh for fixing Nova quotas
Sam Morrison
sorrison at gmail.com
Fri Mar 20 23:53:02 UTC 2015
I put the script up at https://gist.github.com/sorrison/99c8e87295756e0ed787
The cron job that runs is like:
#!/bin/bash
BASEDIR=/tmp
/usr/bin/mysql --defaults-group-suffix=-prod-keystone -B -e 'select target_id, actor_id from assignment inner join project on target_id = project.id inner join user on user.id = actor_id order by target_id, actor_id;' | tail -n+2 > $BASEDIR/keystone_project_users.txt
scp $BASEDIR/keystone_project_users.txt root at nova-controller:.
ssh root at nova-controller nova-manage shell script sync_quotas_main.py
It’s not pretty but it works…..
Sam
> On 21 Mar 2015, at 12:35 am, Kris G. Lindgren <klindgren at godaddy.com> wrote:
>
> Can you post of you cronjob/script that you use to correct the quotas?
> ____________________________________________
>
> Kris Lindgren
> Senior Linux Systems Engineer
> GoDaddy, LLC.
>
>
>
> On 3/20/15, 4:18 AM, "Sam Morrison" <sorrison at gmail.com> wrote:
>
>> We¹ve had the following for a year or so but doesn¹t help much, we still
>> see it occurring every 10 mins or so.
>>
>> max_age = 10
>> until_refresh = 5
>> reservation_expire=600
>>
>> We have a cron job that runs every 10 mins that figures out what projects
>> are out of sync and corrects them.
>> We¹ve always been scared of setting these to zero but we probably should.
>>
>> Sam
>>
>>
>>> On 15 Mar 2015, at 2:53 pm, Mike Dorman <mdorman at godaddy.com> wrote:
>>>
>>> Yeah the default is just Œ0¹ for both, which disables the refresh.
>>>
>>>
>>>
>>> The one downside is that it may not be 100% transparent to the user.
>>> If
>>> the quota is already (incorrectly) too high, and exceeding the quota
>>> limit, the reservation that triggers the refresh will still fail. I.e.
>>> the reservation is attempted based on the quota usage values _before_
>>> the
>>> refresh. But then after that the quota should be fixed and it will
>>> work
>>> again on the next reservation.
>>>
>>> But my thinking is that most quota issues happen slowly over time. If
>>> we
>>> are correcting them often and automatically, they hopefully never get
>>> to
>>> the point where they¹re bad enough to manifest reservation errors to
>>> the
>>> user.
>>>
>>> I don¹t have any information re: db load. I assume it regenerates
>>> based
>>> on what¹s in the instances or reservations table. I imagine the load
>>> for
>>> doing a single refresh is probably comparable to doing a Œnova list¹.
>>>
>>> Mike
>>>
>>>
>>>
>>> On 3/14/15, 2:27 PM, "Tim Bell" <Tim.Bell at cern.ch> wrote:
>>>
>>>> Interesting... what are the defaults ?
>>>>
>>>> Assuming no massive DB load, getting synced within a day would seem
>>>> reasonable. Is the default no max age ?
>>>>
>>>> Tim
>>>>
>>>>> -----Original Message-----
>>>>> From: Jesse Keating [mailto:jlk at bluebox.net]
>>>>> Sent: 14 March 2015 16:59
>>>>> To: openstack-operators at lists.openstack.org
>>>>> Subject: Re: [Openstack-operators] max_age and until_refresh for
>>>>> fixing
>>>>> Nova
>>>>> quotas
>>>>>
>>>>> On 3/14/15 8:11 AM, Mike Dorman wrote:
>>>>>> I did short write-up here http://t.co/Q5X1hTgJG1 if you are
>>>>>> interested
>>>>>> in the details.
>>>>>>
>>>>>
>>>>> Thanks for sharing Matt! That's an excellent write up.
>>>>>
>>>>> --
>>>>> -jlk
>>>>>
>>>>> _______________________________________________
>>>>> OpenStack-operators mailing list
>>>>> OpenStack-operators at lists.openstack.org
>>>>>
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>>
>>>> _______________________________________________
>>>> OpenStack-operators mailing list
>>>> OpenStack-operators at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>> _______________________________________________
>>> OpenStack-operators mailing list
>>> OpenStack-operators at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
More information about the OpenStack-operators
mailing list