Open Stack

I am starting work on a blueprint [1] to create a public API for
subscribing to the message bus in a more generic way than we have for
RPC calls right now. We're already doing this in Ceilometer, and the
Quantum team needs it, too. Unfortunately, the way I ended up doing it
in Ceilometer was with a private method of the Connection.

I've created a wiki page describing the plan [2], but I wanted to
summarize it here on the list and get some more feedback before I
begin implementation.

Ceilometer and Quantum both have some unusual requirements that make
the existing API inadequate:

1. We want to listen for notifications, not RPC calls.

2. We need to listen for notifications from *other projects* not just
   ourselves, so we need to connect to more than one exchange (in AMQP
   parlance).

3. We want to distribute the load among several workers, while
   ensuring that our consumption of notifications does not interrupt
   the flow of those messages to anyone else who also needs them
   (since notifications are a general integration mechanism).

4. While we currently want to listen for notifications, I don't want
   to tie the design to notification-formatted messages so that it
   doesn't work for other non-RPC messages.

My current plann is to create a new method of Connection:

def join_consumer_pool(self, callback, pool_name, topic,
exchange_name=cfg.CONF.control_exchange):
    """Register as a member of a group of consumers for a
    given topic from the specified routing exchange.

    Exactly one member of a given pool will receive each message.

    A message will be delivered to multiple pools, if more than one is
created.
    """

- The callback argument is a callable object to be invoked with a
  single argument, the incoming message (with the JSON unpacked).

- In an AMQP implementation the pool_name argument will be used to
  create or join a queue. Workers in the same pool will use the same
  queue, ensuring that messages can be delivered to more than one
  load-balanced application.

- The topic argument is the usual routing value (i.e., for
  notifications it would be "notifications.info").

- The exchange_name should default to the exchange for the current app
  but passing a value explicitly also allows an application to
  communicate with more than one exchange.

ZMQ-based configurations do not currently have the concept of an
exchange, but after discussing it with Eric we think it would be
useful to add it and pass the value to the matchmaker as a way to
filter the returned clients. I'm not sure yet if we want to use the
term "exchange" or come up with something more generic.

There are more details about some of the implementation details in the
wiki [2], if you're interested in going deeper.

I will start working on the code next week, so please let me know if
you have any concerns with the plan, names, etc.

Thanks,
Doug



[1]
http://wiki.openstack.org/Ceilometer/blueprints/move-listener-framework-oslo
[2]
https://blueprints.launchpad.net/ceilometer/+spec/move-listener-framework-oslo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130208/c9be0e0a/attachment.html>