[openstack-dev] [nova] Distributed Database

Clint Byrum clint at fewbar.com
Thu Apr 28 22:35:15 UTC 2016

Excerpts from Jay Pipes's message of 2016-04-28 13:09:29 -0500:
> On 04/28/2016 08:44 AM, Edward Leafe wrote:
> > On Apr 24, 2016, at 3:28 PM, Robert Collins <robertc at robertcollins.net> wrote:
> >> For instance, the things I think are essential for a distributed
> >> database based datastore:
> >> - good single-machine developer story. Must not need a physical
> >> cluster to hack on OpenStack
> >> - deal gracefully with single node/rack/site failures (when deployed
> >> appropriately) - allow limiting failure domain impact
> >> - straightforward programming model: wrong uses should be obvious to reviewers
> >> - low latency performance with big datasets: e.g. nova list as an
> >> admin should be able to get the Nth page as rapidly as the 2nd or 3rd.
> nova list as an admin (or any user frankly) should be a proxy call to 
> Project Searchlight and elasticsearch.
> elasticsearch is a great interface for this kind of operation. We should 
> use it.
> The Cells architecture, which allows the operator to both scale the 
> message queue *and* limit the scope of failure domains, is a good one. 
> Having a database that stores only the local (to the cell) information 
> is perfectly fine given the top-level API database's index/mapping 
> tables. Where this design has short-comings, as Ed and others point out, 
> are things like doing list operations across dozens of separate cells. 
> So, let's not use the Nova database for that and instead use a solution 
> that works very well for it: Project Searchlight.
> And for those that have small clouds and/or don't want/need to install 
> elasticsearch, OK, cool, stick with a single cell and a single RDBMS 
> instance.

Why are we inventing more things in OpenStack?

- 0MQ allows decentralized RPC and exists today! There's very little
  need for RabbitMQ sharding if it is just handling notifications,
  but if that's a concern, Kafka is also available in oslo.messaging
  and scales out naturally.
  - And now AMQP 1.0 / proton has a legitimate contender for an
    alternative to 0MQ with the dispatch-router approach [1]
  - Or, crazy idea, we could just make RPC happen over http(s).

- Vitess [2] is a proven technology that serves _every_ request to
  Youtube, and provides a familiar SQL interface with sharding built
  in. Shard by project ID and you can just use regular index semantics.
  Or if that's unacceptable (IMO it's fine since Vitess provides enough
  redundancy that one shard has plenty of failure-domain reliability),
  you can also use the built-in Hadoop support they have for doing
  exactly what has been described (merge sorting the result of cross-cell

- If we adopted those, the only reason for cells would be to allow
  setting up new batches of hosts to pre-test before unleashing the
  world on them.  Except, we could do that with host aggregates and
  permissions that hide/expose the right flavors for testing.

So, I have to ask, why is cells v2 being pushed so hard without looking
outside OpenStack for actual existing solutions, which, IMO, are
_numerous_, battle hardened, and simpler than cells.

[1] http://qpid.apache.org/components/dispatch-router/
[2] http://vitess.io/

More information about the OpenStack-dev mailing list