Open Stack

Thanks for having the following warning about those scripts:

'''Since it updates nova DB use it with extremely caution.'''

Anything that doesn't do this transactionally is going to have a hard 
time to keep this correct under active requests (and screwing it up 
means someone's lost $$). Maybe one of these scripts should become an 
official nova tool that is supported by the nova team (until said team 
fixes the root of the scary problem).

Josh

Tim Bell wrote:
> Belmiro has posted the CERN process at http://openstack-in-production.blogspot.fr/2015/03/nova-quota-usage-synchronization.html and code at https://github.com/cernops/nova-quota-sync
>
> Tim
>
>> -----Original Message-----
>> From: Mike Dorman [mailto:mdorman at godaddy.com]
>> Sent: 20 March 2015 18:01
>> To: OpenStack Operators
>> Subject: Re: [Openstack-operators] max_age and until_refresh for fixing Nova
>> quotas
>>
>> Hey Sam,
>>
>> When you say it occurs every 10 minutes, what exactly do you mean?  The quota
>> refresh?  Or a quota getting out of sync?
>>
>> I am surprised you have max_age set so low.  I would think that would basically
>> trigger a quota refresh on every single reservation for most users, right?
>>
>> Mike
>>
>>
>>
>>
>>
>> On 3/20/15, 4:18 AM, "Sam Morrison"<sorrison at gmail.com>  wrote:
>>
>>> We’ve had the following for a year or so but doesn’t help much, we
>>> still see it occurring every 10 mins or so.
>>>
>>> max_age = 10
>>> until_refresh = 5
>>> reservation_expire=600
>>>
>>> We have a cron job that runs every 10 mins that figures out what
>>> projects are out of sync and corrects them.
>>> We’ve always been scared of setting these to zero but we probably should.
>>>
>>> Sam
>>>
>>>
>>>> On 15 Mar 2015, at 2:53 pm, Mike Dorman<mdorman at godaddy.com>
>> wrote:
>>>> Yeah the default is just ‘0’ for both, which disables the refresh.
>>>>
>>>>
>>>>
>>>> The one downside is that it may not be 100% transparent to the user.
>>>> If
>>>> the quota is already (incorrectly) too high, and exceeding the quota
>>>> limit, the reservation that triggers the refresh will still fail.  I.e.
>>>> the reservation is attempted based on the quota usage values _before_
>>>> the  refresh.  But then after that the quota should be fixed and it
>>>> will work  again on the next reservation.
>>>>
>>>> But my thinking is that most quota issues happen slowly over time.
>>>> If we  are correcting them often and automatically, they hopefully
>>>> never get to  the point where they’re bad enough to manifest
>>>> reservation errors to the  user.
>>>>
>>>> I don’t have any information re: db load.  I assume it regenerates
>>>> based  on what’s in the instances or reservations table.  I imagine
>>>> the load for  doing a single refresh is probably comparable to doing a
>>>> ‘nova list’.
>>>>
>>>> Mike
>>>>
>>>>
>>>>
>>>> On 3/14/15, 2:27 PM, "Tim Bell"<Tim.Bell at cern.ch>  wrote:
>>>>
>>>>> Interesting... what are the defaults ?
>>>>>
>>>>> Assuming no massive DB load, getting synced within a day would seem
>>>>> reasonable. Is the default no max age ?
>>>>>
>>>>> Tim
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Jesse Keating [mailto:jlk at bluebox.net]
>>>>>> Sent: 14 March 2015 16:59
>>>>>> To: openstack-operators at lists.openstack.org
>>>>>> Subject: Re: [Openstack-operators] max_age and until_refresh for
>>>>>> fixing  Nova  quotas
>>>>>>
>>>>>> On 3/14/15 8:11 AM, Mike Dorman wrote:
>>>>>>> I did short write-up here http://t.co/Q5X1hTgJG1 if you are
>>>>>>> interested  in the details.
>>>>>>>
>>>>>> Thanks for sharing Matt! That's an excellent write up.
>>>>>>
>>>>>> --
>>>>>> -jlk
>>>>>>
>>>>>> _______________________________________________
>>>>>> OpenStack-operators mailing list
>>>>>> OpenStack-operators at lists.openstack.org
>>>>>>
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operat
>>>>>> ors
>>>>> _______________________________________________
>>>>> OpenStack-operators mailing list
>>>>> OpenStack-operators at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operat
>>>>> ors
>>>> _______________________________________________
>>>> OpenStack-operators mailing list
>>>> OpenStack-operators at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operato
>>>> rs
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators