Open Stack

On 12 April 2016 at 15:48, Andrew Laski <andrew at lascii.com> wrote:

>
>
> On Tue, Apr 5, 2016, at 09:57 AM, Ryan McNair wrote:
> > >It is believed that reservation help to to reserve a set of resources
> > >beforehand and hence eventually preventing any other upcoming request
> > >(serial or parallel) to exceed quota if because of original request the
> > >project might have reached the quota limits.
> > >
> > >Questions :-
> > >1. Does reservation in its current state as used by Nova, Cinder,
> Neutron
> > >help to solve the above problem ?
> >
> > In Cinder the reservations are useful for grouping quota
> > for a single request, and if the request ends up failing
> > the reservation gets rolled back. The reservations also
> > rollback automatically if not committed within a certain
> > time. We also use reservations with Cinder nested quotas
> > to group a usage request that may propagate up to a parent
> > project in order to manage commit/rollback of the request
> > as a single unit.
>

Neutron recently introduced reservations.
Without reservations it was theoretically possible for a tenant to achieve
n times the amount of resources granted by the quota, where n is the number
of workers or distinct server instances.
More informations are available in [1] and [2]


> >
> > >
> > >2. Is it consistent, reliable ?  Even with reservation can we run into
> > >in-consistent behaviour ?
>
>
> > Others can probably answer this better, but I have not
> > seen the reservations be a major issue. In general with
> > quotas we're not doing the check and set atomically which
> > can get us in an inconsistent state with quota-update,
> > but that's unrelated to the reservations.
>

I do not have any news of bugs, nor do I have any know issue that might
affect consistency of the reservation system.
One know weakness has to do with galera clusters - as the reservation
system uses the update lock which is pointless in this case.
Neutron handles the resulting write-set certification failure retrying the
operation, which is quite expensive.
There were already proposals in the nova space to implement a lock-free CAS
algorithm for reservations, but since then I've lost track of developments
in the area.



> >
> > >
> > >3. Do we really need it ?
> > >
> >
> > Seems like we need *some* way of keeping track of usage
> > reserved during a particular request and a way to easily
> > roll that back at a later time. I'm open to alternatives
> > to reservations, just wondering what the big downside of
> > the current reservation system is.
>

Like most things either one proactively ensures a desired condition is met,
or reacts when that condition is not met anymore.
This means that without reservation - eg: optimistic enforcement -
corrective steps must be taken after committing the transaction
that sent the resource over quota. This is completely ok in my opinion. For
instance if taking corrective steps has a cost of 5 and
creating/committing a reservation has a cost of 2, the reactive approach is
convenient if less than 1 request over 3 sends a resource
over quota (note: I've made the numbers up, I just wanted to make a point
that reacting rather than being proactive can be convenient).

However, for Neutron the reactive approach simply won't work because
Neutron leaves a certain degree of freedom to plugins, and several plugins
operate on the backend before committing the DB transaction (I know it's
probably not ok, but if we give them freedom to do so then we cannot
complain I guess). In that case the rollback will be very expensive and it
cannot be a simple DB operation as it has to involve the backend as well.


>
> Jay goes into it a little bit in his response to another quota thread
> http://lists.openstack.org/pipermail/openstack-dev/2016-March/090560.html
> and I share his thoughts here.
>
> With a reservation system you're introducing eventual consistency into
> the system rather than being strict because reservations are not tied to
> a concrete thing. You can't do a point in time check of whether the
> reserved resources are going to eventually be used if something happens
> like a service restart and a request is lost. You have to have no
> activity for the duration of the expiration time to let things settle
> before getting a real view of quota usages.
>

That is true. This is a problem in Neutron that I would like to address too.


>
> Instead if you tie quota usage to the resource records then you can
> always get a view of what's actually in use.
>

Yup, but a reservation and current usage are two different things, aren't
they?


>
> One thing that should probably be clarified in all of these discussion
> is what exactly is the quota on. I see two answers: the quota is against
> the actual resource usage, or the quota is against the records tracking
> usage. Since we currently track quotas with a reservation system I think
> it's fair to say that we're not actually tracking against resource like
> disk/RAM/CPU being in use. I would consider the resource to be in use as
> soon as there's a db record for something like an instance or volume
> which indicates that there will be consumption of those resources. What
> that means in practice is that quotas can be committed right away and
> there doesn't seem to be any need for a reservation system.
>

I think it's fair to say that we track quota against usage counters _and_
reservations.
In my experience with Neutron the performance penalty of doing a count
query versus accessing a record with the usage counter is not that
impressive, and as for not using reservation my comment is that it is
definitely possible, but not necessarily feasible or convenient.

[1] https://bugs.launchpad.net/neutron/+bug/1468934
[2] http://docs.openstack.org/developer/neutron/devref/quota.html

>
>
>
> >
> > - Ryan McNair (mc_nair)
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160414/e582296b/attachment.html>