Open Stack

On Wed, 2014-09-10 at 14:51 +0200, Thierry Carrez wrote:
> Flavio Percoco wrote:
> > [...]
> > Based on the feedback from the meeting[3], the current main concern is:
> > 
> > - Do we need a messaging service with a feature-set akin to SQS+SNS?
> > [...]
> 
> I think we do need, as Samuel puts it, "some sort of durable
> message-broker/queue-server thing". It's a basic application building
> block. Some claim it's THE basic application building block, more useful
> than database provisioning. It's definitely a layer above pure IaaS, so
> if we end up splitting OpenStack into layers this clearly won't be in
> the inner one. But I think "IaaS+" basic application building blocks
> belong in OpenStack one way or another. That's the reason I supported
> Designate ("everyone needs DNS") and Trove ("everyone needs DBs").
> 
> With that said, I think yesterday there was a concern that Zaqar might
> not fill the "some sort of durable message-broker/queue-server thing"
> role well. The argument goes something like: if it was a queue-server
> then it should actually be built on top of Rabbit; if it was a
> message-broker it should be built on top of postfix/dovecot; the current
> architecture is only justified because it's something in between, so
> it's broken.
> 
> I guess I don't mind that much zaqar being "something in between":
> unless I misunderstood, exposing extra primitives doesn't prevent the
> "queue-server" use case from being filled. Even considering the
> message-broker case, I'm also not convinced building it on top of
> postfix/dovecot would be a net win compared to building it on top of
> Redis, to be honest.

AFAICT, this part of the debate boils down to the following argument:

  If Zaqar implemented messaging-as-a-service with only queuing 
  semantics (and no random access semantics), it's design would 
  naturally be dramatically different and "simply" implement a 
  multi-tenant REST API in front of AMQP queues like this:

    https://www.dropbox.com/s/yonloa9ytlf8fdh/ZaqarQueueOnly.png?dl=0

  and that this architecture would allow for dramatically improved 
  throughput for end-users while not making the cost of providing the 
  service prohibitive to operators.

You can't dismiss that argument out-of-hand, but I wonder (a) whether
the claimed performance improvement is going to make a dramatic
difference to the SQS-like use case and (b) whether backing this thing
with an RDBMS and multiple highly available, durable AMQP broker
clusters is going to be too much of a burden on operators for whatever
performance improvements it does gain.

But the troubling part of this debate is where we repeatedly batter the
Zaqar team with hypotheses like these and appear to only barely
entertain their carefully considered justification for their design
decisions like:

  https://wiki.openstack.org/wiki/Frequently_asked_questions_%28Zaqar%29#Is_Zaqar_a_provisioning_service_or_a_data_API.3F
  https://wiki.openstack.org/wiki/Frequently_asked_questions_%28Zaqar%29#What_messaging_patterns_does_Zaqar_support.3F

I would like to see an SQS-like API provided by OpenStack, I accept the
reasons for Zaqar's design decisions to date, I respect that those
decisions were made carefully by highly competent members of our
community and I expect Zaqar to evolve (like all projects) in the years
ahead based on more real-world feedback, new hypotheses or ideas, and
lessons learned from trying things out.

Mark.