Open Stack

On 09/15/2014 09:33 PM, Clint Byrum wrote:
> Excerpts from Zane Bitter's message of 2014-09-15 12:05:09 -0700:
>> On 15/09/14 13:28, Clint Byrum wrote:
>>> Excerpts from Flavio Percoco's message of 2014-09-15 00:57:05 -0700:
>>>> On 09/12/2014 07:13 PM, Clint Byrum wrote:
>>>>> Excerpts from Thierry Carrez's message of 2014-09-12 02:16:42 -0700:
>>>>>> Clint Byrum wrote:
>>>>>>> Excerpts from Flavio Percoco's message of 2014-09-11 04:14:30 -0700:
>>>>>>>> Is Zaqar being optimized as a *queuing* service? I'd say no. Our goal is
>>>>>>>> to optimize Zaqar for delivering messages and supporting different
>>>>>>>> messaging patterns.
>>>>>>>
>>>>>>> Awesome! Just please don't expect people to get excited about it for
>>>>>>> the lighter weight queueing workloads that you've claimed as use cases.
>>>>>>>
>>>>>>> I totally see Horizon using it to keep events for users. I see Heat
>>>>>>> using it for stack events as well. I would bet that Trove would benefit
>>>>>>> from being able to communicate messages to users.
>>>>>>>
>>>>>>> But I think in between Zaqar and the backends will likely be a lighter
>>>>>>> weight queue-only service that the users can just subscribe to when they
>>>>>>> don't want an inbox. And I think that lighter weight queue service is
>>>>>>> far more important for OpenStack than the full blown random access
>>>>>>> inbox.
>>>>>>>
>>>>>>> I think the reason such a thing has not appeared is because we were all
>>>>>>> sort of running into "but Zaqar is already incubated". Now that we've
>>>>>>> fleshed out the difference, I think those of us that need a lightweight
>>>>>>> multi-tenant queue service should add it to OpenStack.  Separately. I hope
>>>>>>> that doesn't offend you and the rest of the excellent Zaqar developers. It
>>>>>>> is just a different thing.
>>>>>>>
>>>>>>>> Should we remove all the semantics that allow people to use Zaqar as a
>>>>>>>> queue service? I don't think so either. Again, the semantics are there
>>>>>>>> because Zaqar is using them to do its job. Whether other folks may/may
>>>>>>>> not use Zaqar as a queue service is out of our control.
>>>>>>>>
>>>>>>>> This doesn't mean the project is broken.
>>>>>>>
>>>>>>> No, definitely not broken. It just isn't actually necessary for many of
>>>>>>> the stated use cases.
>>>>>>
>>>>>> Clint,
>>>>>>
>>>>>> If I read you correctly, you're basically saying the Zaqar is overkill
>>>>>> for a lot of people who only want a multi-tenant queue service. It's
>>>>>> doing A+B. Why does that prevent people who only need A from using it ?
>>>>>>
>>>>>> Is it that it's actually not doing A well, from a user perspective ?
>>>>>> Like the performance sucks, or it's missing a key primitive ?
>>>>>>
>>>>>> Is it that it's unnecessarily complex to deploy, from a deployer
>>>>>> perspective, and that something only doing A would be simpler, while
>>>>>> covering most of the use cases?
>>>>>>
>>>>>> Is it something else ?
>>>>>>
>>>>>> I want to make sure I understand your objection. In the "user
>>>>>> perspective" it might make sense to pursue both options as separate
>>>>>> projects. In the "deployer perspective" case, having a project doing A+B
>>>>>> and a project doing A doesn't solve anything. So this affects the
>>>>>> decision we have to take next Tuesday...
>>>>>
>>>>> I believe that Zaqar does two things, inbox semantics, and queue
>>>>> semantics. I believe the queueing is a side-effect of needing some kind
>>>>> of queue to enable users to store and subscribe to messages in the
>>>>> inbox.
>>>>>
>>>>> What I'd rather see is an API for queueing, and an API for inboxes
>>>>> which integrates well with the queueing API. For instance, if a user
>>>>> says "give me an inbox" I think Zaqar should return a queue handle for
>>>>> sending into the inbox the same way Nova gives you a Neutron port if
>>>>> you don't give it one. You might also ask for a queue to receive push
>>>>> messages from the inbox. Point being, the queues are not the inbox,
>>>>> and the inbox is not the queues.
>>>>>
>>>>> However, if I just want a queue, just give me a queue. Don't store my
>>>>> messages in a randomly addressable space, and don't saddle the deployer
>>>>> with the burden of such storage. Put the queue API in front of a scalable
>>>>> message queue and give me a nice simple HTTP API. Users would likely be
>>>>> thrilled. Heat, Nova, Ceilometer, probably Trove and Sahara, could all
>>>>> make use of just this. Only Horizon seems to need a place to keep the
>>>>> messages around while users inspect them.
>>>>>
>>>>> Whether that is two projects, or one, separation between the two API's,
>>>>> and thus two very different types of backends, is something I think
>>>>> will lead to more deployers wanting to deploy both, so that they can
>>>>> bill usage appropriately and so that their users can choose wisely.
>>>>
>>>> This is one of the use-cases we designed flavors for. One of the mail
>>>> ideas behind flavors is giving the user the choice of where they want
>>>> their messages to be stored. This certainly requires the deployer to
>>>> have installed stores that are good for each job. For example, based on
>>>> the current existing drivers, a deployer could have configured a
>>>> high-throughput flavor on top of a redis node that has been configured
>>>> to perform for this job. Alongside to this flavor, the deployer could've
>>>> configured a flavor that features durability on top of mongodb or redis.
>>>>
>>>> When the user creates the queue/bucket/inbox/whatever they want to put
>>>> their messages into, they'll be able to choose where those messages
>>>> should be stored into based on their needs.
>>>>
>>>> I do understand your objection is not against Zaqar being able to do
>>>> this now or not but whether an integrate API for both kind of semantics
>>>> makes sense or not. I still fail to see why they need to be separated,
>>>> though. Billing can still happen properly based on the resource usage,
>>>> flavors and other metrics.
>>>
>>> As long as it can be built into the API that certain flavors don't allow
>>> pagination of messages, I can see it working.
>>
>> If the v2 API removes random access to messages, is there any reason for 
>> it to allow listing of messages at all? AFAICT the v2 API could 
>> eliminate any access to or knowledge about anything but the first 
>> unclaimed message in the queue while still satisfying all of the target 
>> use cases.
>>
> 
> I don't think that handles Horizon's case. The notifications need to
> stick around, and be acknowledged by the user, and that may not happen
> in order. If it doesn't happen, then the messages actually need to be
> available from the queue again.
> 
>>> Also maybe some flavors
>>> are auto-ACK and thus can be backed entirely by AMQP/gearman/etc. without
>>> any special gymnastics to support sessions.
>>
>> That doesn't make a lot of sense to me - if you have an extremely high 
>> volume of messages and you don't care about losing some, just spin up 
>> your own AMQP broker on a Nova server. Zaqar offers guaranteed delivery 
>> of messages, which at very small scales can only be economically 
>> provided by the cloud operator. Users who don't care about either of 
>> those things have a host of other good, cheap options already. If you 
>> just want someone else to administer that for you, then you get the 
>> Trove-style queue-provisioning service that was mooted at various points 
>> in this thread.
>>
> 
> The issue is how do you ACK messages in AMQP transactions without the
> active session where the message came from? Since we have a stateless
> HTTP protocol, we need something to maintain that specific AMQP session
> until the HTTP protocol receives an ACK from the client after the client
> has consumed or somehow stored the message. So, these queues have to be
> auto-ACK, or we have to figure out a way to have something maintain the
> session.

Auto-ACK was one of the options we had to support AMQP brokers as a
backend but it didn't feel right and we didn't have flavors so it felt
even worse :P.

Now that we have flavors, if we put a broker with auto-ACK behind zaqar,
we'll be basically using the broker as a read-once database and that
seems like a waste to me. We're basically hiding/blocking/wasting all
the other broker features at the cost of having to maintain it and all
the other problems brokers bring in.

What would be the real benefit of using say, rabbit, configured with
aut-ACK as a backend for Zaqar's messaging API?


>>> However, I tend to think that if one thing is a superset of the other,
>>> they can also be separate, and typically in a scale-out context, if
>>> something can be separate, it should be.
>>
>> AIUI the Zaqar API processes are essentially stateless and defer the 
>> storage to some other system encapsulated behind a storage plugin. Given 
>> that the storage parts already scale out separately, it doesn't follow 
>> at all that they should have separate APIs for scale-out reasons.
>>
> 
> If we drop the Horizon use case, I agree.
> 
> However, if Horizon still wants to show the user all the notifications,
> out of order, and paginated, and ack'd in different orders (read: the
> inbox use case) then that is, IMO, an entirely separate concern that
> would basically be the v1 API.

I'm not sure why we should deprecate Horizon's use case. Lets start by
saying that horizon's use case is not fully covered just yet but why
would it have to do it out of order? Also, is the ack really necessary?


pub/sub doesn't necessarily guarantees messages delivery, it really
depends on the implementation. That said, there are ways to guarantee
that depending on the method used. For example, if the subscriber is a
webhook, we can use the response status code to ack the message. if it
has a persistent connection like websocket or even (long|short)-poll an
ack may be needed.

That said, I don't see why we should drop Horizon's use case.

Could you elaborate a bit more?

Thanks :)
Flavio


-- 
@flaper87
Flavio Percoco