Open Stack

At some point in the near future, hopefully early in L, we're intending
to update Nova to use the new database transaction management in
oslo.db's enginefacade.

Spec:
http://git.openstack.org/cgit/openstack/oslo-specs/plain/specs/kilo/make-enginefacade-a-facade.rst

Implementation:
https://review.openstack.org/#/c/138215/

One of the effects of this is that we will always know when we are in a
read-only transaction, or a transaction which includes writes. We intend
to use this new contextual information to make greater use of read-only
slave databases. We are currently proposing that if an admin has
configured a slave database, we will use the slave for *all* read-only
transactions. This would make the use_slave parameter passed to some
Nova apis redundant, as we would always use the slave where the context
allows.

However, using a slave database has a potential pitfall when mixed with
separate write transactions. A caller might currently:

1. start a write transaction
2. update the database
3. commit the transaction
4. start a read transaction
5. read from the database

The client might expect data written in step 2 to be reflected in data
read in step 5. I can think of 3 cases here:

1. A short-lived RPC call is using multiple transactions

This is a bug which the new enginefacade will help us eliminate. We
should not be using multiple transactions in this case. If the reads are
in the same transaction as the write: they will be on the master, they
will be consistent, and there is no problem. As a bonus, lots of these
will be race conditions, and we'll fix at least some.

2. A long-lived task is using multiple transactions between long-running
sub-tasks

In this case, for example creating a new instance, we genuinely want
multiple transactions: we don't want to hold a database transaction open
while we copy images around. However, I can't immediately think of a
situation where we'd write data, then subsequently want to read it back
from the db in a read-only transaction. I think we will typically be
updating state, meaning it's going to be a succession of write transactions.

3. Separate RPC calls from a remote client

This seems potentially problematic to me. A client makes an RPC call to
create a new object. The client subsequently tries to retrieve the
created object, and gets a 404.

Summary: 1 is a class of bugs which we should be able to find fairly
mechanically through unit testing. 2 probably isn't a problem in
practise? 3 seems like a problem, unless consumers of cloud services are
supposed to expect that sort of thing.

I understand that slave databases can occasionally get very behind. How
behind is this in practise?

How do we use use_slave currently? Why do we need a use_slave parameter
passed in via rpc, when it should be apparent to the developer whether a
particular task is safe for out-of-date data.

Any chance they have some kind of barrier mechanism? e.g. block until
the current state contains transaction X.

General comments on the usefulness of slave databases, and the
desirability of making maximum use of them?

Thanks,

Matt
-- 
Matthew Booth
Red Hat Engineering, Virtualisation Team

Phone: +442070094448 (UK)
GPG ID:  D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490