Open Stack

Excerpts from Flavio Percoco's message of 2014-09-15 00:57:05 -0700:
> On 09/12/2014 07:13 PM, Clint Byrum wrote:
> > Excerpts from Thierry Carrez's message of 2014-09-12 02:16:42 -0700:
> >> Clint Byrum wrote:
> >>> Excerpts from Flavio Percoco's message of 2014-09-11 04:14:30 -0700:
> >>>> Is Zaqar being optimized as a *queuing* service? I'd say no. Our goal is
> >>>> to optimize Zaqar for delivering messages and supporting different
> >>>> messaging patterns.
> >>>
> >>> Awesome! Just please don't expect people to get excited about it for
> >>> the lighter weight queueing workloads that you've claimed as use cases.
> >>>
> >>> I totally see Horizon using it to keep events for users. I see Heat
> >>> using it for stack events as well. I would bet that Trove would benefit
> >>> from being able to communicate messages to users.
> >>>
> >>> But I think in between Zaqar and the backends will likely be a lighter
> >>> weight queue-only service that the users can just subscribe to when they
> >>> don't want an inbox. And I think that lighter weight queue service is
> >>> far more important for OpenStack than the full blown random access
> >>> inbox.
> >>>
> >>> I think the reason such a thing has not appeared is because we were all
> >>> sort of running into "but Zaqar is already incubated". Now that we've
> >>> fleshed out the difference, I think those of us that need a lightweight
> >>> multi-tenant queue service should add it to OpenStack.  Separately. I hope
> >>> that doesn't offend you and the rest of the excellent Zaqar developers. It
> >>> is just a different thing.
> >>>
> >>>> Should we remove all the semantics that allow people to use Zaqar as a
> >>>> queue service? I don't think so either. Again, the semantics are there
> >>>> because Zaqar is using them to do its job. Whether other folks may/may
> >>>> not use Zaqar as a queue service is out of our control.
> >>>>
> >>>> This doesn't mean the project is broken.
> >>>
> >>> No, definitely not broken. It just isn't actually necessary for many of
> >>> the stated use cases.
> >>
> >> Clint,
> >>
> >> If I read you correctly, you're basically saying the Zaqar is overkill
> >> for a lot of people who only want a multi-tenant queue service. It's
> >> doing A+B. Why does that prevent people who only need A from using it ?
> >>
> >> Is it that it's actually not doing A well, from a user perspective ?
> >> Like the performance sucks, or it's missing a key primitive ?
> >>
> >> Is it that it's unnecessarily complex to deploy, from a deployer
> >> perspective, and that something only doing A would be simpler, while
> >> covering most of the use cases?
> >>
> >> Is it something else ?
> >>
> >> I want to make sure I understand your objection. In the "user
> >> perspective" it might make sense to pursue both options as separate
> >> projects. In the "deployer perspective" case, having a project doing A+B
> >> and a project doing A doesn't solve anything. So this affects the
> >> decision we have to take next Tuesday...
> > 
> > I believe that Zaqar does two things, inbox semantics, and queue
> > semantics. I believe the queueing is a side-effect of needing some kind
> > of queue to enable users to store and subscribe to messages in the
> > inbox.
> > 
> > What I'd rather see is an API for queueing, and an API for inboxes
> > which integrates well with the queueing API. For instance, if a user
> > says "give me an inbox" I think Zaqar should return a queue handle for
> > sending into the inbox the same way Nova gives you a Neutron port if
> > you don't give it one. You might also ask for a queue to receive push
> > messages from the inbox. Point being, the queues are not the inbox,
> > and the inbox is not the queues.
> > 
> > However, if I just want a queue, just give me a queue. Don't store my
> > messages in a randomly addressable space, and don't saddle the deployer
> > with the burden of such storage. Put the queue API in front of a scalable
> > message queue and give me a nice simple HTTP API. Users would likely be
> > thrilled. Heat, Nova, Ceilometer, probably Trove and Sahara, could all
> > make use of just this. Only Horizon seems to need a place to keep the
> > messages around while users inspect them.
> > 
> > Whether that is two projects, or one, separation between the two API's,
> > and thus two very different types of backends, is something I think
> > will lead to more deployers wanting to deploy both, so that they can
> > bill usage appropriately and so that their users can choose wisely.
> 
> This is one of the use-cases we designed flavors for. One of the mail
> ideas behind flavors is giving the user the choice of where they want
> their messages to be stored. This certainly requires the deployer to
> have installed stores that are good for each job. For example, based on
> the current existing drivers, a deployer could have configured a
> high-throughput flavor on top of a redis node that has been configured
> to perform for this job. Alongside to this flavor, the deployer could've
> configured a flavor that features durability on top of mongodb or redis.
> 
> When the user creates the queue/bucket/inbox/whatever they want to put
> their messages into, they'll be able to choose where those messages
> should be stored into based on their needs.
> 
> I do understand your objection is not against Zaqar being able to do
> this now or not but whether an integrate API for both kind of semantics
> makes sense or not. I still fail to see why they need to be separated,
> though. Billing can still happen properly based on the resource usage,
> flavors and other metrics.

As long as it can be built into the API that certain flavors don't allow
pagination of messages, I can see it working. Also maybe some flavors
are auto-ACK and thus can be backed entirely by AMQP/gearman/etc. without
any special gymnastics to support sessions.

However, I tend to think that if one thing is a superset of the other,
they can also be separate, and typically in a scale-out context, if
something can be separate, it should be.