[openstack-dev] [marconi] Reconsidering the unified API model

Doug Hellmann doug.hellmann at dreamhost.com
Mon Jun 9 21:19:17 UTC 2014

On Mon, Jun 9, 2014 at 3:31 PM, Kurt Griffiths
<kurt.griffiths at rackspace.com> wrote:
> Folks, this may be a bit of a bombshell, but I think we have been dancing
> around the issue for a while now and we need to address it head on. Let me
> start with some background.
> Back when we started designing the Marconi API, we knew that we wanted to
> support several messaging patterns. We could do that using a unified queue
> resource, combining both task distribution and feed semantics. Or we could
> create disjoint resources in the API, or even create two separate services
> altogether, one each for the two semantic groups.
> The decision was made to go with a unified API for these reasons:
> It would afford hybrid patterns, such as auditing or diagnosing a task
> distribution queue
> Once you implement guaranteed delivery for a message feed over HTTP,
> implementing task distribution is a relatively straightforward addition. If
> you want both types of semantics, you don’t necessarily gain anything by
> implementing them separately.
> Lately we have been talking about writing drivers for traditional message
> brokers that will not be able to support the message feeds part of the API.
> I’ve started to think that having a huge part of the API that may or may not
> “work”, depending on how Marconi is deployed, is not a good story for users,
> esp. in light of the push to make different clouds more interoperable.
> Therefore, I think we have a very big decision to make here as a team and a
> community. I see three options right now. I’ve listed several—but by no
> means conclusive—pros and cons for each, as well as some counterpoints,
> based on past discussions.
> Option A. Allow drivers to only implement part of the API
> For:
> Allows for a wider variety of backends. (counter: may create subtle
> differences in behavior between deployments)
> May provide opportunities for tuning deployments for specific workloads
> Against:
> Makes it hard for users to create applications that work across multiple
> clouds, since critical functionality may or may not be available in a given
> deployment. (counter: how many users need cross-cloud compatibility? Can
> they degrade gracefully?)
> Option B. Split the service in two. Different APIs, different services. One
> would be message feeds, while the other would be something akin to Amazon’s
> SQS.
> For:
> Same as Option A, plus creates a clean line of functionality for deployment
> (deploy one service or the other, or both, with clear expectations of what
> messaging patterns are supported in any case).
> Against:
> Removes support for hybrid messaging patterns (counter: how useful are such
> patterns in the first place?)
> Operators now have two services to deploy and support, rather than just one
> (counter: can scale them independently, perhaps leading to gains in
> efficiency)
> Option C. Require every backend to support the entirety of the API as it now
> stands.
> For:
> Least disruptive in terms of the current API design and implementation
> Affords a wider variety of messaging patterns (counter: YAGNI?)
> Reuses code in drivers and API between feed and task distribution operations
> (counter: there may be ways to continue sharing some code if the API is
> split)
> Against:
> Requires operators to deploy a NoSQL cluster (counter: many operators are
> comfortable with NoSQL today)
> Currently requires MongoDB, which is AGPL (counter: a Redis driver is under
> development)
> A unified API is hard to tune for performance (counter: Redis driver should
> be able to handle high-throughput use cases, TBD)

We went with a single large storage API in ceilometer initially, but
we had some discussions at the Juno summit about it being a bad
decision because it resulted in storing some data like alarm
definitions in database formats that just didn't make sense for that.
Julien and Eoghan may want to fill in more details.

Keystone has separate backends for tenants, tokens, the catalog, etc.,
so you have precedent there for splitting up the features in a way
that makes it easier for driver authors and for building features on
appropriate backends.


> I’d love to get everyone’s thoughts on these options; let's brainstorm for a
> bit, then we can home in on the option that makes the most sense. We may
> need to do some POCs or experiments to get enough information to make a good
> decision.
> @kgriffs
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

More information about the OpenStack-dev mailing list