[openstack-dev] [Quantum][Common] Private rpc methods / declare_topic_consumer

Doug Hellmann doug.hellmann at dreamhost.com
Fri Sep 7 11:12:07 UTC 2012

On Thu, Sep 6, 2012 at 12:25 PM, Eric Windisch <eric at cloudscaling.com>wrote:

> > We needed something like declare_topic_consumer() in ceilometer and I
> couldn't find another way to listen to notifications without using the AMQP
> libraries directly. It's on my list of things to add to the RPC library,
> but I'm still working on getting ceilometer working end-to-end, so I
> haven't started circling back and "fixing" stuff like this, yet. I hope to
> have more of that done in grizzly.
> >
> Right. The question is: does Quantum change what it is doing to follow a
> standard pattern, as you have, or do we add something like
> declare_topic_consumer() to the abstraction?  I'm thinking the latter might
> be a good thing to have, regardless. I acknowledge this would require
> implementation of this in the ZeroMQ driver and I realistically can't
> expect this to land in Folsom.  In a way, changing the Quantum behavior is
> easier to scope.

I agree the right way to fix this from a design perspective is to update
the abstraction. I can't speak for the level of work needed to change
Quantum or update ZeroMQ.

> >
> > I'm not sure what you mean about removing the queue name argument.
> That's there so the client can indicate which messages it wants. Do you
> have in mind another way to do that?
> >
> You're saying that you're assuming the topic is an exchange, and allowing
> the queue name to be something else?

That's how it works for AMQP. Several workers join the same queue so that
only one of them receives a given message. The topic controls the messages
sent to that queue. Multiple worker pools can use different queue names so
that each pool receives a copy of the message. That allows the same
horizontal-scaling pattern as the declare_worker() method does for full RPC
messages and is a critical part of the design of ceilometer, where we are
expecting to be processing a large number of messages with a pool of
several collector daemons.

>  I can see the value in that, especially in the private methods.  I'd
> still be inclined to starting with a generalized declare_topic_consumer()
> being without a queue_name argument in a first-pass, it is far less
> complex.  Long-term, this needs to be handled by the MatchMaker and the
> ServiceGroupAPI - at least in the ZeroMQ driver - much of which won't land
> until Grizzly.  The generalized form could pass into the more flexible (but
> private) declare_topic_consumer() in the case of Kombu and RabbitMQ.  Later
> in Grizzly, if we can evolve the matchmaker and ServiceGroupAPI
> sufficiently, then we might add an equivilent to the queue_name argument to
> the generalized form.  I hope this is reasonable?

It's fine to update the method argument names to improve the abstraction,
but we need both arguments to be passed separately for the method to be
useful in ceilometer.

Is there documentation for the MatchMaker or ServiceGroupAPI somewhere?
I've seen them mentioned before, but gather that they are relatively new
and I haven't had a chance to dig into them at all.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20120907/0960e443/attachment-0001.html>

More information about the OpenStack-dev mailing list