Open Stack

I am investigating using ZMQ for our openstack implementation because
we have been having trouble using Rabbit in distributed, multimaster
mode.  One of our design goals is to achieve HA via giving all nodes
access to all system state and round-robining all incoming requests
via HAProxy.

ZMQ makes sense from a decentralization standpoint, but I have some
questions about the design decisions around it. I'm asking in hopes
that someone can explain some nuances of OpenStack that I may be
missing, as I'm relatively new to it.

My first question is: Why the matchmaker?   ZMQ supports pub/sub quite
swimmingly and the current mechanism for fetching the queues list
seems like an unnecessary (albeit small) hack.

A fully connected graph is needed, which means a single publisher
won't suffiice.  This means that each node who wishes to receive
broadcasts must connect to all publishers.  In most fully
decentrialized systems I've worked in, there is a notion of a quorum
of network membership, and this is generally a queryable or locally
maintained (and fully connected) service.

Why the distinction?  I guess it only matters if you're interested in
adding and removing participating nodes at run time, especially nodes
which might not be performing all openstack tasks, but only a subset.
Then it is up to only that node to subscribe to the topics that he is
after.  In the Matchmaker approach, a node that is joining the network
would be responsible for ensuring that all local matchmakers are
updated with the topics in which it is interested.

The local network membership service seems like something that would
be useful for a variety of tasks, so perhaps it could be lifted out .

I guess the more abstract question is: What is the appetite for
OpenStack to move to a fully decentrialized model? Given OpenStack's
origins in Twisted, it seems like it would be a design goal. However,
currently it is relying on  abstractions such as MessageQueues and
Databases to provide federation, but, in my experience, there aren't
many AMQP or RDBMS technologies out there that can perform reliably in
a fully decentralized system.

Apologies for the rambling email.

Rick Richardson