Open Stack

Great job highlighting what our friends over at Amazon are doing.

It's clear from these snippets, and a few other pieces of documentation
for SQS I've read, that the Amazon team approached SQS from a _massive_
scaling perspective. I think what may be forcing a lot of this frustration
with Zaqar is that it was designed with a much smaller scale in mind.

I think as long as that is the case, the design will remain in question.
I'd be comfortable saying that the use cases I've been thinking about
are entirely fine with the limitations SQS has.

Excerpts from Joe Gordon's message of 2014-09-17 13:36:18 -0700:
> Hi All,
> 
> My understanding of Zaqar is that it's like SQS. SQS uses distributed
> queues, which have a few unusual properties [0]:
> Message Order
> 
> Amazon SQS makes a best effort to preserve order in messages, but due to
> the distributed nature of the queue, we cannot guarantee you will receive
> messages in the exact order you sent them. If your system requires that
> order be preserved, we recommend you place sequencing information in each
> message so you can reorder the messages upon receipt.
> At-Least-Once Delivery
> 
> Amazon SQS stores copies of your messages on multiple servers for
> redundancy and high availability. On rare occasions, one of the servers
> storing a copy of a message might be unavailable when you receive or delete
> the message. If that occurs, the copy of the message will not be deleted on
> that unavailable server, and you might get that message copy again when you
> receive messages. Because of this, you must design your application to be
> idempotent (i.e., it must not be adversely affected if it processes the
> same message more than once).
> Message Sample
> 
> The behavior of retrieving messages from the queue depends whether you are
> using short (standard) polling, the default behavior, or long polling. For
> more information about long polling, see Amazon SQS Long Polling
> <http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html>
> .
> 
> With short polling, when you retrieve messages from the queue, Amazon SQS
> samples a subset of the servers (based on a weighted random distribution)
> and returns messages from just those servers. This means that a particular
> receive request might not return all your messages. Or, if you have a small
> number of messages in your queue (less than 1000), it means a particular
> request might not return any of your messages, whereas a subsequent request
> will. If you keep retrieving from your queues, Amazon SQS will sample all
> of the servers, and you will receive all of your messages.
> 
> The following figure shows short polling behavior of messages being
> returned after one of your system components makes a receive request.
> Amazon SQS samples several of the servers (in gray) and returns the
> messages from those servers (Message A, C, D, and B). Message E is not
> returned to this particular request, but it would be returned to a
> subsequent request.
> 
> 
> 
> Presumably SQS has these properties because it makes the system scalable,
> if so does Zaqar have the same properties (not just making these same
> guarantees in the API, but actually having these properties in the
> backends)? And if not, why? I looked on the wiki [1] for information on
> this, but couldn't find anything.
> 
> 
> 
> 
> 
> [0]
> http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/DistributedQueues.html
> [1] https://wiki.openstack.org/wiki/Zaqar