Open Stack

> Hi All,
> 
> My understanding of Zaqar is that it's like SQS. SQS uses distributed queues,
> which have a few unusual properties [0]:
> Message Order
> 
> 
> Amazon SQS makes a best effort to preserve order in messages, but due to the
> distributed nature of the queue, we cannot guarantee you will receive
> messages in the exact order you sent them. If your system requires that
> order be preserved, we recommend you place sequencing information in each
> message so you can reorder the messages upon receipt.
> At-Least-Once Delivery
> 
> 
> Amazon SQS stores copies of your messages on multiple servers for redundancy
> and high availability. On rare occasions, one of the servers storing a copy
> of a message might be unavailable when you receive or delete the message. If
> that occurs, the copy of the message will not be deleted on that unavailable
> server, and you might get that message copy again when you receive messages.
> Because of this, you must design your application to be idempotent (i.e., it
> must not be adversely affected if it processes the same message more than
> once).
> Message Sample
> 
> 
> The behavior of retrieving messages from the queue depends whether you are
> using short (standard) polling, the default behavior, or long polling. For
> more information about long polling, see Amazon SQS Long Polling .
> 
> With short polling, when you retrieve messages from the queue, Amazon SQS
> samples a subset of the servers (based on a weighted random distribution)
> and returns messages from just those servers. This means that a particular
> receive request might not return all your messages. Or, if you have a small
> number of messages in your queue (less than 1000), it means a particular
> request might not return any of your messages, whereas a subsequent request
> will. If you keep retrieving from your queues, Amazon SQS will sample all of
> the servers, and you will receive all of your messages.
> 
> The following figure shows short polling behavior of messages being returned
> after one of your system components makes a receive request. Amazon SQS
> samples several of the servers (in gray) and returns the messages from those
> servers (Message A, C, D, and B). Message E is not returned to this
> particular request, but it would be returned to a subsequent request.
> 
> 
> 
> 
> 
> 
> 
> Presumably SQS has these properties because it makes the system scalable, if
> so does Zaqar have the same properties (not just making these same
> guarantees in the API, but actually having these properties in the
> backends)? And if not, why? I looked on the wiki [1] for information on
> this, but couldn't find anything.

The premise of this thread is flawed I think.

It seems to be predicated on a direct quote from the public
documentation of a closed-source system justifying some
assumptions about the internal architecture and design goals
of that closed-source system.

It then proceeds to hold zaqar to account for not making
the same choices as that closed-source system.

This puts the zaqar folks in a no-win situation, as it's hard
to refute such arguments when they have no visibility over
the innards of that closed-source system.

Sure, the assumption may well be correct that the designers
of SQS made the choice to expose applications to out-of-order
messages as this was the only practical way of acheiving their
scalability goals.

But since the code isn't on github and the design discussions
aren't publicly archived, we have no way of validating that.

Would it be more reasonable to compare against a cloud-scale
messaging system that folks may have more direct knowledge
of?

For example, is HP Cloud Messaging[1] rolled out in full
production by now?

Is it still cloning the original Marconi API, or has it kept
up with the evolution of the API? Has the nature of this API
been seen as the root cause of any scalability issues?

Cheers,
Eoghan

[1] http://www.openstack.org/blog/2013/05/an-introductory-tour-of-openstack-cloud-messaging-as-a-service