Open Stack

Folks, this may be a bit of a bombshell, but I think we have been dancing around the issue for a while now and we need to address it head on. Let me start with some background.

Back when we started designing the Marconi API, we knew that we wanted to support several messaging patterns. We could do that using a unified queue resource, combining both task distribution and feed semantics. Or we could create disjoint resources in the API, or even create two separate services altogether, one each for the two semantic groups.

The decision was made to go with a unified API for these reasons:

  *   It would afford hybrid patterns, such as auditing or diagnosing a task distribution queue
  *   Once you implement guaranteed delivery for a message feed over HTTP, implementing task distribution is a relatively straightforward addition. If you want both types of semantics, you don’t necessarily gain anything by implementing them separately.

Lately we have been talking about writing drivers for traditional message brokers that will not be able to support the message feeds part of the API. I’ve started to think that having a huge part of the API that may or may not “work”, depending on how Marconi is deployed, is not a good story for users, esp. in light of the push to make different clouds more interoperable.

Therefore, I think we have a very big decision to make here as a team and a community. I see three options right now. I’ve listed several—but by no means conclusive—pros and cons for each, as well as some counterpoints, based on past discussions.

Option A. Allow drivers to only implement part of the API

For:

  *   Allows for a wider variety of backends. (counter: may create subtle differences in behavior between deployments)
  *   May provide opportunities for tuning deployments for specific workloads

Against:

  *   Makes it hard for users to create applications that work across multiple clouds, since critical functionality may or may not be available in a given deployment. (counter: how many users need cross-cloud compatibility? Can they degrade gracefully?)

Option B. Split the service in two. Different APIs, different services. One would be message feeds, while the other would be something akin to Amazon’s SQS.

For:

  *   Same as Option A, plus creates a clean line of functionality for deployment (deploy one service or the other, or both, with clear expectations of what messaging patterns are supported in any case).

Against:

  *   Removes support for hybrid messaging patterns (counter: how useful are such patterns in the first place?)
  *   Operators now have two services to deploy and support, rather than just one (counter: can scale them independently, perhaps leading to gains in efficiency)

Option C. Require every backend to support the entirety of the API as it now stands.

For:

  *   Least disruptive in terms of the current API design and implementation
  *   Affords a wider variety of messaging patterns (counter: YAGNI?)
  *   Reuses code in drivers and API between feed and task distribution operations (counter: there may be ways to continue sharing some code if the API is split)

Against:

  *   Requires operators to deploy a NoSQL cluster (counter: many operators are comfortable with NoSQL today)
  *   Currently requires MongoDB, which is AGPL (counter: a Redis driver is under development)
  *   A unified API is hard to tune for performance (counter: Redis driver should be able to handle high-throughput use cases, TBD)

I’d love to get everyone’s thoughts on these options; let's brainstorm for a bit, then we can home in on the option that makes the most sense. We may need to do some POCs or experiments to get enough information to make a good decision.

@kgriffs
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140609/09459004/attachment.html>