Open Stack

On Thu, Apr 25, 2013 at 4:39 PM, Doug Hellmann
<doug.hellmann at dreamhost.com>wrote:

>
>
>
> On Thu, Apr 25, 2013 at 4:06 PM, Eric Windisch <eric at cloudscaling.com>wrote:
>
>> >
>> > What about making all of those arguments except topic part of the
>> constructor for a Connection, and just instantiating more than one of them
>> to talk to different services?
>> If we can figure out what they are, or how to make this per-driver and
>> store them in a reasonable way - I think we might be on the right path.
>> >
>> >
>> >
>> > It seems like the host and port options for rabbit and qpid map to the
>> same values for the matchmaker, don't they? The fact that the actual
>> message doesn't go to the matchmaker is an implementation detail of ZMQ
>> that it can hide.
>> The matchmaker is pluggable. Some matchmakers use a flat file or local
>> database, not a network service (although this may practically become a
>> requirement for a network service, in the long run?).  There may not be a
>> single IP or DNS based endpoint for this, it may be multiple hosts, or a
>> multicast address for communications, etc. I think it is premature to
>> suggest that all the backends will have a host/port/user/password
>> associated, and that other configuration fields won't be necessary.
>>
>
> OK, I was thinking of something like this:
>
> # in openstack.common.rpc
>
> def make_connection_factory(config, rpc_driver_name):
>     driver = load_driver(rpc_driver_name)
>     conn_factory = driver.make_connection_factory(config)
>     return conn_factory
>

After thinking about this a little more I realized that this doesn't solve
the problem of allowing one service to talk to another on a separate
"exchange." If that's still something we want to do, we need another
argument passed to the ConnectionFactory to represent that exchange
explicitly (I know we need another name, but until we come up with one I'll
stick with "exchange"). The default can come from the config, but callers
like ceilometer need a way to specify an alternative value.

Do we care if the exchange for a service is not on the same host? Do we
need to allow users to provide different host/port settings for every
exchange?

Doug


>
> # in the driver
>
> class ConnectionFactory:
>     def __init__(self, config):
>        # know how to find useful configuration settings for the driver,
>        # like the amqp server or the location of the matchmaker.
>
>     def __call__(self, host):
>        # driver-specific logic for combining "host" with the values pulled
> in __init_
>        # to have enough connection parameters
>        return Connection(those_parameters)
>
> # in the app
>
> cf = make_connection_factory(cfg.CONF, cfg.CONF.rpc.driver_name)
> c = cf(host)
> response = c.call(topic, message)
>
>
>>
>> Also, I think that at least optionally, the matchmaker might become tied
>> to the AMQP drivers. That might be a good thing, at least in leveling the
>> field. The more we talk about message encryption, and in some models, even
>> for signing, we might find having the matchmaker advantageous. TBD.
>> > Right, it seems like the host portion needs to be passed as a separate
>> argument and the AMQP drivers can combine it with the topic, instead of
>> going the other way around.
>> I'm suggesting we let "service.host" be a valid topic, but *also* pass
>> the host variable into the call/cast so that drivers can do what they will
>> with it. It won't be *necessary* for the topic to be called "service.host",
>> but it won't be illegal.
>>
>
> I'd rather not mix and match. If the host value is just a way to limit the
> recipient of the message going to topic (or to have the sender choose
> instead of message broker), then the fact that we define a separate queue
> for that is an implementation detail of the driver, just like the fact that
> the ZMQ driver doesn't have a broker, but is always communicating
> point-to-point.
>
>
>>
>> > > > For ZeroMQ, a control_exchange is effectively a zmq_port *AND* its
>> > > > associated matchmaker, which is pluggable. Plugins usually will have
>> > > > their own host/port/auth. The MatchMaker basically maintains a
>> mapping
>> > > > of topics to consuming peers. Technically, we could lookup all
>> topics
>> > > > here, even "compute.host" topics, but we can circumvent a global
>> > > > lookup by sending messages directly to "host".
>> > >
>> > >
>> > > Understood (because well explained, thanks :)
>> >
>> > How does the ZMQ driver know where the host is without asking the
>> matchmaker? Is that "host" value assumed to be resolvable via DNS?
>> Either because it infers it from the call/cast command (presently by
>> inspecting the topic - which is no longer a valid source of authority), or
>> if there is no host to be inferred, by looking up in the matchmaker.
>>
>> The host itself is whatever is passed as CONF.rpc_zmq_host, which MUST
>> match whatever Nova's CONF.host is set to. This defaults to the FQDN, such
>> that DNS resolution is required. In *practice*, we override both of these
>> variables to the system's IP address (for the interface on which messaging
>> should happen)
>>
>
> OK, so host is intended to be the sort of thing you could pass as part of
> an address when creating a socket. If we make the host argument to
> ConnectionFactory.__call__ optional then the topic values can be consistent
> (i.e., not include the host) and the caller can still control who receives
> the message by constructing the Connection appropriately. That way the
> semantics of the call are controlled by the right arguments. The topic says
> "the message is about this" and the host limits the "audience" that gets
> the message.
>
>
>> >
>> >
>> > > > Perhaps an rpc-abstraction level concept of a "cell" or "project" is
>> > > > needed? That would be an identifier encompassing all the connection
>> > > > details (i.e. host/control_exchange, zmq_port/matchmaker), however
>> > > > such may be defined for the driver being used?
>> > >
>> > >
>> > > That sounds like a good idea to me. That's what I called "connection"
>> in
>> > > my previous mail actually. This may be just an abstract class
>> > > implemented by each driver in its own way.
>> >
>> >
>> > What defines a "cell" for RPC? Just the alternate host and port for
>> rabbit, qpid, or matchmaker? If so, then the caller just needs to create a
>> new Connection with the right settings, and we don't need to have the
>> concept of "cell" leak into the RPC library API at all. It's there now
>> because we're using a global connection object, but sometimes want to talk
>> to something different. If we stop having a global, the awkwardness will go
>> away and the API can be simpler.
>> You have this basically correct. Part of the problem is that we don't
>> have that abstraction currently, and the parameters passed from the cell
>> code for host/etc are really ill-matching for the MatchMaker requirements…
>> and confused with the unclear separation of zmq_port and control_exchange
>> for ZeroMQ -- and a question if ZeroMQ messages should, or should not, be
>> brokered through to child cells (which may be necessary for some network
>> security purposes, but unnecessary for queue scaling)
>>
>> What I'd like to have, ideally, is the cell configuration essentially
>> reconfigure the ZeroMQ settings as necessary, per the deployer's
>> preference. A brokered-to-cell solution wouldn't be available, at least not
>> presently, but such a design would allow that to be a possible feature
>> addition/request, should it be requested.  (I also don't want to bike shed
>> or overdesign too much on what people *might* do or want down the road)
>>
>
> Given the fact that we have completely different calls right now, it seems
> like the caller must already know when it is talking to a cell.
> Instantiating a separate Connection for those cases should be fine, as long
> as we can find the matchmaker for that cell using the configuration data
> passed to make_connection_factory(). Maybe we need a cell_name argument to
> that call?
>
> Doug
>
>
>>
>> Regards,
>> Eric Windisch
>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130425/8fdec496/attachment.html>