Open Stack

I ran the basics of our new distributed architecture by Shawn from
Twitter.  Here's his response.


<Tim> We just held a Congress mid-cycle meet-up to discuss its distributed
architecture.  We decided on an architecture where the policy engine runs
in its own process; each datasource driver runs in its own process; the API
runs in its own process; and all communicate over a RabbitMQ-style message
bus.

This would enable you to choose which part of Congress you run on which
physical host.  You could run the policy engine on the same box as the
service you're using policy to protect.  You can build the datasource
driver logic into a datasource and push deltas directly to the RabbitMQ
message bus.  And of course you could run everything on a
Congress-dedicated box and ignore these details.

See any major problems with Congress's new distributed architecture, or
things you think should change?  Do you think that running the policy
engine on the same box as the service you're protecting is a good idea or a
bad one?

<Shawn>I suspect RabbitMQ may end up being a bottleneck but it makes sense
to deal with that when it comes up. Will you be relying on
RabbitMQ-specific features or will any given AMQP broker suffice? Which
version of AMQP will you be using?


<Tim>OpenStack has a wrapper around AMQP buses, and we're using that.  We
haven't gone any further into choices about which plugin to use.

<Shawn>I think the choice of where to run the datasource makes sense,
though I need to give some thought to wiring up my datasources directly to
the AMQP broker instead of submitting via a well-defined REST or thrift API
with realtime payload validation, backpressure, etc. I can't tell whether
my hesitation is legit or just FUD.

Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150902/0caf91df/attachment.html>