[openstack-dev] [nova][database][quotas] reservations table ??
afazekas at redhat.com
Mon Apr 13 15:10:53 UTC 2015
----- Original Message -----
> From: "Kevin L. Mitchell" <kevin.mitchell at rackspace.com>
> To: openstack-dev at lists.openstack.org
> Sent: Friday, April 10, 2015 5:47:26 PM
> Subject: Re: [openstack-dev] [nova][database][quotas] reservations table ??
> On Fri, 2015-04-10 at 02:38 -0400, Attila Fazekas wrote:
> > I noticed the nova DB has reservations table with an expire field (+24h)
> > and a periodic task
> > in the scheduler (60 sec) for expire the otherwise not deleted records .
> > Both the table and the observed operations are strange.
> > What this table and its operations are trying to solve ?
> > Why does it needed ?
> > Why this solution was chosen ?
> It might help to know that this is reservations for the quota system.
> The basic reason that this exists is because of parallelism: say the
> user makes a request to boot a new instance, and that new instance would
> fill their quota. Nova begins processing the request, but while it's
> doing so, the user makes a second (or third, fourth, fifth, etc.)
> request. With a reservation, we can count the first request against
> their quota and reject the extra requests; without a reservation, we
> have no way of knowing that nova is already processing a request, and so
> could allow the user to vastly exceed their quota.
Just the very existence of the `expire` makes the solution very suspicious.
As I see the operations does no ensure parallel safe quota enforcement
at resource creation and based on stale data. (wireshark)
It is based on a data originated from different transaction,
even without SELECT .. WITH SHARED LOCK.
When moving the delta to/from reservations the service puts a lock
(SELECT .. FOR UPDATE) on all same tenant related quota_usages row,
this is the only safety mechanism I saw.
Alone it is not enough.
No quota related table touched in the same transaction
when the instance state changed (or created). :(
The reservations table is not really needed.
What is really needed is doing the quota_usages changes
and resource state changes in the same transaction !
The transactions are all or nothing constructs,
nothing can happen which needs any `expire` thing.
The transaction needs to ensure really it does the state change.
It can mean just read it with SELECT .. FOR UPDATE
for an existing record (for ex.: instance)
The transaction also needs to ensure quota check happened
based on not stale data -> SELECT .. WITH SHARED LOCK for
- quota limit queries
- for calculating the actual number of things or for just reading the
values from the quota_usages
In most cases, the quota check and update can be merged to a single UPDATE statement
and it fully can happen on the DB side, without actually fetching
any quota related information by the service.
The mysql UPDATE statement with the right expressions and sub-queries,
automatically can place the minimum required locks and do the update when needed.
The number of changed rows returned by the UPDATE,
can indicate is the quota successfully allocated (passed the check) or not.
When it's not successful, just ROLLBACK and tell something to the user about
the `Out of Quota` issue.
It is recommended to put the quota check close to the end of the transaction,
in order to minimize the lock hold time related to quota_usages table.
At the end we will not lock the quota_usages twice (as we do now),
and we do not left behind 4 virtually deleted rows in a `bonus` table,
and do not use +1 extra transaction and +8 extra UPDATE per instance create,
and consistency is ensured.
> > PS.:
> > Is the uuid in the table referenced by anything?
> Once the operation that allocated the reservation completes, it either
> rolls back the reservation (in the case of failure) or it commits the
> reservation (updating a cache quota usages table). This involves
> updating the reservation table to delete the reservation, and a UUID
> helps match up the specific row. (Or rows; most operations involve more
> than one quota and thus more than one row.) The expiration logic is to
> deal with the case that the operation never completed because nova
> crashed in the middle, and provides a stop-gap measure to ensure that
> the usage isn't counted against the user forever.
Just to confirm, the same UUID just exists in the reservation table only,
and temporary in one workers memory .?
> Kevin L. Mitchell <kevin.mitchell at rackspace.com>
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
The `Refresh` is also strange thing in this context.
More information about the OpenStack-dev