Open Stack

Excerpts from Flavio Percoco's message of 2014-09-11 04:14:30 -0700:
> On 09/10/2014 03:45 PM, Gordon Sim wrote:
> > On 09/10/2014 01:51 PM, Thierry Carrez wrote:
> >> I think we do need, as Samuel puts it, "some sort of durable
> >> message-broker/queue-server thing". It's a basic application building
> >> block. Some claim it's THE basic application building block, more useful
> >> than database provisioning. It's definitely a layer above pure IaaS, so
> >> if we end up splitting OpenStack into layers this clearly won't be in
> >> the inner one. But I think "IaaS+" basic application building blocks
> >> belong in OpenStack one way or another. That's the reason I supported
> >> Designate ("everyone needs DNS") and Trove ("everyone needs DBs").
> >>
> >> With that said, I think yesterday there was a concern that Zaqar might
> >> not fill the "some sort of durable message-broker/queue-server thing"
> >> role well. The argument goes something like: if it was a queue-server
> >> then it should actually be built on top of Rabbit; if it was a
> >> message-broker it should be built on top of postfix/dovecot; the current
> >> architecture is only justified because it's something in between, so
> >> it's broken.
> > 
> > What is the distinction between a message broker and a queue server? To
> > me those terms both imply something broadly similar (message broker
> > perhaps being a little bit more generic). I could see Zaqar perhaps as
> > somewhere between messaging and data-storage.
> 
> I agree with Gordon here. I really don't know how to say this without
> creating more confusion. Zaqar is a messaging service. Messages are the
> most important entity in Zaqar. This, however, does not forbid anyone to
> use Zaqar as a queue. It has the required semantics, it guarantees FIFO
> and other queuing specific patterns. This doesn't mean Zaqar is trying
> to do something outside its scope, it comes for free.
> 

It comes with a huge cost actually, so saying it comes for free is a
misrepresentation. It is a side effect of developing a superset of
queueing. But that superset is only useful to a small number of your
stated use cases. Many of your use cases (including the one I've been
involved with, Heat pushing metadata to servers) are entirely served by
the much simpler, much lighter weight, pure queueing service.

> Is Zaqar being optimized as a *queuing* service? I'd say no. Our goal is
> to optimize Zaqar for delivering messages and supporting different
> messaging patterns.
> 

Awesome! Just please don't expect people to get excited about it for
the lighter weight queueing workloads that you've claimed as use cases.

I totally see Horizon using it to keep events for users. I see Heat
using it for stack events as well. I would bet that Trove would benefit
from being able to communicate messages to users.

But I think in between Zaqar and the backends will likely be a lighter
weight queue-only service that the users can just subscribe to when they
don't want an inbox. And I think that lighter weight queue service is
far more important for OpenStack than the full blown random access
inbox.

I think the reason such a thing has not appeared is because we were all
sort of running into "but Zaqar is already incubated". Now that we've
fleshed out the difference, I think those of us that need a lightweight
multi-tenant queue service should add it to OpenStack.  Separately. I hope
that doesn't offend you and the rest of the excellent Zaqar developers. It
is just a different thing.

> Should we remove all the semantics that allow people to use Zaqar as a
> queue service? I don't think so either. Again, the semantics are there
> because Zaqar is using them to do its job. Whether other folks may/may
> not use Zaqar as a queue service is out of our control.
> 
> This doesn't mean the project is broken.
> 

No, definitely not broken. It just isn't actually necessary for many of
the stated use cases.