Open Stack

Hi,
I just posted in our operations blog how CERN is dealing with quotas
synchronization problem.

http://openstack-in-production.blogspot.fr/2015/03/nova-quota-usage-synchronization.html

Hope it helps,
cheers,
Belmiro



On Sat, Mar 21, 2015 at 12:55 AM, Sam Morrison <sorrison at gmail.com> wrote:

> I’d need to go through the logs to see exactly the occurrence maybe 10
> minutes was a bit much. We run the cron job every 10 mins.
>
> The cron job stopped a few weeks back and we had at least half a dozen
> projects with out of sync quotas within a few hours.
>
> Cheers,
> Sam
>
>
> > On 21 Mar 2015, at 4:01 am, Mike Dorman <mdorman at godaddy.com> wrote:
> >
> > Hey Sam,
> >
> > When you say it occurs every 10 minutes, what exactly do you mean?  The
> > quota refresh?  Or a quota getting out of sync?
> >
> > I am surprised you have max_age set so low.  I would think that would
> > basically trigger a quota refresh on every single reservation for most
> > users, right?
> >
> > Mike
> >
> >
> >
> >
> >
> > On 3/20/15, 4:18 AM, "Sam Morrison" <sorrison at gmail.com> wrote:
> >
> >> We’ve had the following for a year or so but doesn’t help much, we still
> >> see it occurring every 10 mins or so.
> >>
> >> max_age = 10
> >> until_refresh = 5
> >> reservation_expire=600
> >>
> >> We have a cron job that runs every 10 mins that figures out what
> projects
> >> are out of sync and corrects them.
> >> We’ve always been scared of setting these to zero but we probably
> should.
> >>
> >> Sam
> >>
> >>
> >>> On 15 Mar 2015, at 2:53 pm, Mike Dorman <mdorman at godaddy.com> wrote:
> >>>
> >>> Yeah the default is just ‘0’ for both, which disables the refresh.
> >>>
> >>>
> >>>
> >>> The one downside is that it may not be 100% transparent to the user.
> >>> If
> >>> the quota is already (incorrectly) too high, and exceeding the quota
> >>> limit, the reservation that triggers the refresh will still fail.  I.e.
> >>> the reservation is attempted based on the quota usage values _before_
> >>> the
> >>> refresh.  But then after that the quota should be fixed and it will
> >>> work
> >>> again on the next reservation.
> >>>
> >>> But my thinking is that most quota issues happen slowly over time.  If
> >>> we
> >>> are correcting them often and automatically, they hopefully never get
> >>> to
> >>> the point where they’re bad enough to manifest reservation errors to
> >>> the
> >>> user.
> >>>
> >>> I don’t have any information re: db load.  I assume it regenerates
> >>> based
> >>> on what’s in the instances or reservations table.  I imagine the load
> >>> for
> >>> doing a single refresh is probably comparable to doing a ‘nova list’.
> >>>
> >>> Mike
> >>>
> >>>
> >>>
> >>> On 3/14/15, 2:27 PM, "Tim Bell" <Tim.Bell at cern.ch> wrote:
> >>>
> >>>> Interesting... what are the defaults ?
> >>>>
> >>>> Assuming no massive DB load, getting synced within a day would seem
> >>>> reasonable. Is the default no max age ?
> >>>>
> >>>> Tim
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Jesse Keating [mailto:jlk at bluebox.net]
> >>>>> Sent: 14 March 2015 16:59
> >>>>> To: openstack-operators at lists.openstack.org
> >>>>> Subject: Re: [Openstack-operators] max_age and until_refresh for
> >>>>> fixing
> >>>>> Nova
> >>>>> quotas
> >>>>>
> >>>>> On 3/14/15 8:11 AM, Mike Dorman wrote:
> >>>>>> I did short write-up here http://t.co/Q5X1hTgJG1 if you are
> >>>>>> interested
> >>>>>> in the details.
> >>>>>>
> >>>>>
> >>>>> Thanks for sharing Matt! That's an excellent write up.
> >>>>>
> >>>>> --
> >>>>> -jlk
> >>>>>
> >>>>> _______________________________________________
> >>>>> OpenStack-operators mailing list
> >>>>> OpenStack-operators at lists.openstack.org
> >>>>>
> >>>>>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >>>>
> >>>> _______________________________________________
> >>>> OpenStack-operators mailing list
> >>>> OpenStack-operators at lists.openstack.org
> >>>>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >>> _______________________________________________
> >>> OpenStack-operators mailing list
> >>> OpenStack-operators at lists.openstack.org
> >>>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >>
> > _______________________________________________
> > OpenStack-operators mailing list
> > OpenStack-operators at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20150321/72bd12da/attachment.html>