Open Stack

On 09/17/2014 10:36 PM, Joe Gordon wrote:
> Hi All,
> 
> My understanding of Zaqar is that it's like SQS. SQS uses distributed
> queues, which have a few unusual properties [0]:
> 
> 
>     Message Order
> 
> Amazon SQS makes a best effort to preserve order in messages, but due to
> the distributed nature of the queue, we cannot guarantee you will
> receive messages in the exact order you sent them. If your system
> requires that order be preserved, we recommend you place sequencing
> information in each message so you can reorder the messages upon receipt.
> 

Zaqar guarantees FIFO. To be more precise, it does that relying on the
storage backend ability to do so as well. Depending on the storage used,
guaranteeing FIFO may have some performance penalties.

We have discussed on adding a way to enable/disable some features - like
FIFO - in a per-queue bases and now that we have flavors, we may work on
allowing things like enabling/disabling FIFO on a per-flavor basis.


>     At-Least-Once Delivery
> 
> Amazon SQS stores copies of your messages on multiple servers for
> redundancy and high availability. On rare occasions, one of the servers
> storing a copy of a message might be unavailable when you receive or
> delete the message. If that occurs, the copy of the message will not be
> deleted on that unavailable server, and you might get that message copy
> again when you receive messages. Because of this, you must design your
> application to be idempotent (i.e., it must not be adversely affected if
> it processes the same message more than once).

As of now, Zaqar fully relies on the storage replication/clustering
capabilities to provide data consistency, availability and fault
tolerance. However, as far as consuming messages is concerned, it can
guarantee once-and-only-once and/or at-least-once delivery depending on
the message pattern used to consume messages. Using pop or claims
guarantees the former whereas streaming messages out of Zaqar guarantees
the later.


>     Message Sample
> 
> The behavior of retrieving messages from the queue depends whether you
> are using short (standard) polling, the default behavior, or long
> polling. For more information about long polling, see Amazon SQS Long
> Polling
> <http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html>.
> 
> With short polling, when you retrieve messages from the queue, Amazon
> SQS samples a subset of the servers (based on a weighted random
> distribution) and returns messages from just those servers. This means
> that a particular receive request might not return all your messages.
> Or, if you have a small number of messages in your queue (less than
> 1000), it means a particular request might not return any of your
> messages, whereas a subsequent request will. If you keep retrieving from
> your queues, Amazon SQS will sample all of the servers, and you will
> receive all of your messages.
> 
> The following figure shows short polling behavior of messages being
> returned after one of your system components makes a receive request.
> Amazon SQS samples several of the servers (in gray) and returns the
> messages from those servers (Message A, C, D, and B). Message E is not
> returned to this particular request, but it would be returned to a
> subsequent request.

Image here:
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/images/ArchOverview_Receive.png

Currently the best and only way to pull messages from Zaqar is by using
short polling. We'll work on long-polling and websocket support.
Requests are guaranteed to return available messages if there are any.

> Presumably SQS has these properties because it makes the
> system scalable, if so does Zaqar have the same properties (not just
> making these same guarantees in the API, but actually having these
> properties in the backends)? And if not, why?

Based on our short conversation on IRC last night, I understand you're
concerned that FIFO may result in performance issues. That's a valid
concern and I think the right answer is that it depends on the storage.
If the storage has a built-in FIFO guarantee then there's nothing Zaqar
needs to do there. In the other hand, if the storage does not have a
built-in support for FIFO, Zaqar will cover it in the driver
implementation. In the mongodb driver, each message has a marker that is
used to guarantee FIFO.


> I looked on the wiki [1]
> for information on this, but couldn't find anything.
> 


We have this[0] section in the FAQ but it definitely doesn't answer your
questions. I'm sure we had this documented way better but I can't find
the link. :(


[0]
https://wiki.openstack.org/wiki/Zaqar/Frequently_asked_questions#How_does_Zaqar_compare_to_AWS_.28SQS.2FSNS.29.3F

Hope the above helps,
Flavio

-- 
@flaper87
Flavio Percoco