<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 19, 2014 at 4:23 AM, Eoghan Glynn <span dir="ltr"><<a href="mailto:eglynn@redhat.com" target="_blank">eglynn@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><br>
<br>
> Hi All,<br>
><br>
> My understanding of Zaqar is that it's like SQS. SQS uses distributed queues,<br>
> which have a few unusual properties [0]:<br>
> Message Order<br>
><br>
><br>
> Amazon SQS makes a best effort to preserve order in messages, but due to the<br>
> distributed nature of the queue, we cannot guarantee you will receive<br>
> messages in the exact order you sent them. If your system requires that<br>
> order be preserved, we recommend you place sequencing information in each<br>
> message so you can reorder the messages upon receipt.<br>
> At-Least-Once Delivery<br>
><br>
><br>
> Amazon SQS stores copies of your messages on multiple servers for redundancy<br>
> and high availability. On rare occasions, one of the servers storing a copy<br>
> of a message might be unavailable when you receive or delete the message. If<br>
> that occurs, the copy of the message will not be deleted on that unavailable<br>
> server, and you might get that message copy again when you receive messages.<br>
> Because of this, you must design your application to be idempotent (i.e., it<br>
> must not be adversely affected if it processes the same message more than<br>
> once).<br>
> Message Sample<br>
><br>
><br>
> The behavior of retrieving messages from the queue depends whether you are<br>
> using short (standard) polling, the default behavior, or long polling. For<br>
> more information about long polling, see Amazon SQS Long Polling .<br>
><br>
> With short polling, when you retrieve messages from the queue, Amazon SQS<br>
> samples a subset of the servers (based on a weighted random distribution)<br>
> and returns messages from just those servers. This means that a particular<br>
> receive request might not return all your messages. Or, if you have a small<br>
> number of messages in your queue (less than 1000), it means a particular<br>
> request might not return any of your messages, whereas a subsequent request<br>
> will. If you keep retrieving from your queues, Amazon SQS will sample all of<br>
> the servers, and you will receive all of your messages.<br>
><br>
> The following figure shows short polling behavior of messages being returned<br>
> after one of your system components makes a receive request. Amazon SQS<br>
> samples several of the servers (in gray) and returns the messages from those<br>
> servers (Message A, C, D, and B). Message E is not returned to this<br>
> particular request, but it would be returned to a subsequent request.<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> Presumably SQS has these properties because it makes the system scalable, if<br>
> so does Zaqar have the same properties (not just making these same<br>
> guarantees in the API, but actually having these properties in the<br>
> backends)? And if not, why? I looked on the wiki [1] for information on<br>
> this, but couldn't find anything.<br>
<br>
</div></div>The premise of this thread is flawed I think.<br>
<br>
It seems to be predicated on a direct quote from the public<br>
documentation of a closed-source system justifying some<br>
assumptions about the internal architecture and design goals<br>
of that closed-source system.<br>
<br>
It then proceeds to hold zaqar to account for not making<br>
the same choices as that closed-source system.<br>
<br>
This puts the zaqar folks in a no-win situation, as it's hard<br>
to refute such arguments when they have no visibility over<br>
the innards of that closed-source system.<br>
<br>
Sure, the assumption may well be correct that the designers<br>
of SQS made the choice to expose applications to out-of-order<br>
messages as this was the only practical way of acheiving their<br>
scalability goals.<br>
<br>
But since the code isn't on github and the design discussions<br>
aren't publicly archived, we have no way of validating that.<br>
<br>
Would it be more reasonable to compare against a cloud-scale<br>
messaging system that folks may have more direct knowledge<br>
of?<br>
<br>
For example, is HP Cloud Messaging[1] rolled out in full<br>
production by now?<br>
<br></blockquote><div><br></div><div>Unfortunately the HP Cloud Messaging service was decommissioned. </div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Is it still cloning the original Marconi API, or has it kept<br>
up with the evolution of the API? Has the nature of this API<br>
been seen as the root cause of any scalability issues?<br>
<br></blockquote><div><br></div><div>We created a RabbitMQ backed implementation that aimed to be API compatible with Marconi. This proved difficult given some of the API issues that have been discussed on this very thread. Our implementation could never be full API compatible with Marconi (there really isn’t an easy way to map AMQP to HTTP, without losing serious functionality).</div><div><br></div><div>We also worked closely with the Marconi team, trying to get upstream to support AMQP — the Marconi team also came to the same conclusion that their API was not a good fit for such a backend.</div><div><br></div><div>Now — we are looking at options. One that intrigues us has also been suggested on these threads, specifically building a ‘managed messaging service’ that could provision various messaging technologies (rabbit, kafka, etc), and at the end of the day hand off the protocol native to the messaging technology to the end user. </div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Cheers,<br>
Eoghan<br>
<br>
[1] <a href="http://www.openstack.org/blog/2013/05/an-introductory-tour-of-openstack-cloud-messaging-as-a-service" target="_blank">http://www.openstack.org/blog/2013/05/an-introductory-tour-of-openstack-cloud-messaging-as-a-service</a><br>
<div><div><br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>