[openstack-dev] [oslo.messaging][zeromq] pub/sub support for the revised zeromq driver

Alec Hothan (ahothan) ahothan at cisco.com
Thu Jun 18 17:31:25 UTC 2015



On 6/1/15, 5:03 PM, "Davanum Srinivas" <davanum at gmail.com<mailto:davanum at gmail.com>> wrote:

fyi, the spec for zeromq driver in oslo.messaging is here:
https://review.openstack.org/#/c/187338/1/specs/liberty/zmq-patterns-usage
.rst,unified


The above spec suggests using the zmq pub/sub/xpub/xsub proxy pattern for implementing the oslo messaging fanout API.
This is described in the zmq guide (http://zguide.zeromq.org/, search for "Figure 13 - Pub-Sub Network with a Proxy").

When applied to openstack, this pattern will result in a number of publishers connecting to a
potentially large number of proxies (the proposal calls for 1 proxy per node in the cloud), and each proxy connected to a relatively small number of local services (typically
agents).

All connections in this pattern are based on TCP (for inter-node) and could be based on IPC (Unix domain socket) for intra-node.
ZMQ provides the option of having either side initiate the connection, in our case, it is logical and simple to have the agents connect to their
local proxy (just use any well known unix domain socket name). However the connectivity on the other end (publisher to proxy) is not as simple without some form of name service/service discovery (in the present version of ZMQ driver, this is handled by matchmaker/redis).

GROUP MANAGEMENT
The ZMQ guide mentions rightly so that this pattern is great for decoupling publishers and subscribers as they can independently connect to the proxy without having to know each other.
So publishers would basically simply connect to *the* proxy and not have to deal with other publishers or any subscriber.
However when the number of proxies can be very large (thousands), the difficulty resides in keeping track of nodes as they can get added or removed during regular operations. When a node is added, we would need a way to
have each publisher connect to that new node.  Same can be said about node deletion.
If we go that route we will need a good solution on how to do that efficiently and at scale.

And this is one of my main reserve for the spec currently proposed in gerrit.

To handle group management, we could reuse the existing matchmaker code (based on redis) or decide to use a more streamlined solution based on a name service. I would like to hear about any feedback on the use of matchmaker at scale if there is any experience (because I do have lots of questions on how matchmaker is designed and how it fares regarding scale and HA).

I'll try to start another email thread regarding scale numbers (number of connections, current usage of fanout in the code, emulated vs. true multicast)

  Alec
















-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150618/a1cd989b/attachment.html>


More information about the OpenStack-dev mailing list