Open Stack

On 18/09/14 10:55, Flavio Percoco wrote:
> On 09/18/2014 04:24 PM, Clint Byrum wrote:
>> Great job highlighting what our friends over at Amazon are doing.
>>
>> It's clear from these snippets, and a few other pieces of documentation
>> for SQS I've read, that the Amazon team approached SQS from a _massive_
>> scaling perspective. I think what may be forcing a lot of this frustration
>> with Zaqar is that it was designed with a much smaller scale in mind.
>>
>> I think as long as that is the case, the design will remain in question.
>> I'd be comfortable saying that the use cases I've been thinking about
>> are entirely fine with the limitations SQS has.
>
> I think these are pretty strong comments with not enough arguments to
> defend them.

I actually more or less agree with Clint here. As Joe noted (and *thank 
you* Joe for starting this thread - the first one to compare Zaqar to 
something relevant!), SQS offers very, very limited guarantees, and it's 
clear that the reason for that is to make it massively, massively 
scalable in the way that e.g. S3 is scalable while also remaining 
comparably durable (S3 is supposedly designed for 11 nines, BTW).

Zaqar, meanwhile, seems to be promising the world in terms of 
guarantees. (And then taking it away in the fine print, where it says 
that the operator can disregard many of them, potentially without the 
user's knowledge.)

On the other hand, IIUC Zaqar does in fact have a sharding feature 
("Pools") which is its answer to the massive scaling question. I don't 
know enough details to comment further except to say that it evidently 
has been carefully thought about at some level, and it's really 
frustrating for the Zaqar folks when people just assume that it hasn't 
without doing any research. On the face of it sharding is a decent 
solution for this problem. Maybe we need to dig into the details and 
make sure folks are satisfied that there are no hidden dragons.

> Saying that Zaqar was designed with a smaller scale in mid without
> actually saying why you think so is not fair besides not being true. So
> please, do share why you think Zaqar was not designed for big scales and
> provide comments that will help the project to grow and improve.
>
> - Is it because the storage technologies that have been chosen?
> - Is it because of the API?
> - Is it because of the programing language/framework ?

I didn't read Clint and Devananda's comments as an attack on any of 
these things (although I agree that there have been far too many such 
attacks over the last 12 months from people who didn't bother to do 
their homework first). They're looking at Zaqar from first principles 
and finding that it promises too much, raising the concern the team may 
in future reach a point where they are unable to meet the needs of 
future users (perhaps for scaling reasons) without breaking existing 
users who have come to rely on those promises.

> So far, we've just discussed the API semantics and not zaqar's
> scalability, which makes your comments even more surprising.

What guarantees you offer can determine pretty much all of the design 
tradeoffs (e.g. latency vs. durability) that you have to make. Some of 
those (e.g. random access to messages) are baked in to the API, but 
others are not. It's actually a real concern to me to see elsewhere in 
this thread that you are punting to operators on many of the latter.

IMO the Zaqar team needs to articulate an opinionated vision of just 
what Zaqar actually is, and why it offers value. And by 'value' here I 
mean there should be $ signs attached.

For example, it makes no sense to me that Zaqar should ever be able to 
run in a mode that doesn't guarantee delivery of messages. There are a 
million and one easy, cheap ways to set up a system that _might_ deliver 
your message. One server running a message broker is sufficient. But if 
you want reliable delivery, then you'll probably need at least 3 (for 
still pretty low values of "reliable"). I did some back-of-the-envelope 
math with the AWS pricing and _conservatively_ for any application 
receiving <100k messages per hour (~30 per second) it's cheaper to use 
SQS than to spin up those servers yourself.

In other words, a service that *guarantees* delivery of messages *has* 
to be run by the cloud operator because for the overwhelming majority of 
applications, the user cannot do so economically.

(I'm assuming here that AWS's _relative_ pricing accurately reflects 
their _relative_ cost basis, which is almost certainly not strictly 
true, but I expect a close enough approximation for these purposes.)

What I would like to hear in this thread is:

"Zaqar is We-Never-Ever-Ever-Ever-Lose-Your-Message as a Service 
(WNEEELYMaaS), and it has to be in OpenStack because only the cloud 
operator can provide that cost-effectively."

What I'm hearing instead is:

- We'll probably deliver your message.
- We can guarantee that we'll deliver your message, but only on clouds 
where the operator has chosen to configure Mongo with some non-default 
setting, and it's really all up to them.
- We have "Flavors" so your app can be broken in a different way on 
every cloud you port it to.
- When your client goes down for any reason, all bets are off (in some 
modes).
- At least the messages will be in order!

I encourage the Zaqar core team to share their vision for what Zaqar is 
(I know they have one, and I know what I said above isn't it), and IMHO 
that vision should be along the lines of "it's orders of magnitude more 
likely that your entire region will go down than that you'll ever lose a 
message, on _any_ cloud (at any scale) where Zaqar runs, even when your 
client dies while processing it". Everything, and I mean *everything*, 
else - FIFO, pub-sub, low latency, browsability - is gravy. Tasty, tasty 
gravy in some cases (FIFO in particular seems like a potential 
game-changer for many applications), but if it impacts durability, 
scalability or deployability it should be gone.

Right now the vision that is coming through is "Zaqar is whatever sort 
of messaging the operator wants it to be". Trying to please everyone is 
a common trap for Open Source projects to fall into (and I think the TC 
has to take a large measure of blame, for promoting multiple competing 
visions for what Zaqar should be in lieu of trying to understand the 
original one), but I don't think it can be allowed to fly here. Whether 
your messaging system is reliable or unreliable in large part determines 
the design of your application. If the answer is different on each 
cloud, then the answer is 'unreliable' (on multiple levels), and 
unreliable messaging is not something that needs to be in OpenStack 
because users can implement it themselves with little difficulty, at 
relatively low cost and with much more flexibility.


However, unlike (I suspect) some people in this thread, I don't believe 
that major technical changes are required to Zaqar to achieve what we 
want. Zaqar's design is already very flexible and can probably handle 
just about any back-end architecture we choose for it. I would 
personally love to see a Swift back-end, since that would solve the 
deployment-of-another-architecture problem, the 
semantics-depend-on-operator-configuration problem and the 
massive-scalability problem at a stroke, but I don't think this is a 
critical priority. The Zaqar team can be trusted to do that sort of 
implementation work if and when makes sense. What *is* critical is that 
we lock down the guarantees that Zaqar provides to the user to the 
smallest useful set, chisel them in stone, and explicitly disclaim all 
other guarantees.

Only by doing that can we be sure that Zaqar will have the freedom to 
move to other back-ends, should they be deemed more appropriate, without 
breaking users. There will be some minor tweaks to the API for v2 to 
achieve that, but a lot of it is about just setting expectations.

This thread has been really helpful because we're discussing the 
semantics presented to the user from first principles, and I think the 
way forward is to collect those together in one place and debate which 
are essential and which can - and, it almost follows, should - be dropped.


As an aside, in thinking about this I have for the first time identified 
a criterion by which I would be prepared to see all new projects judged. 
Some members of the TC have suggested that projects must be able to 
point to existing, large-scale deployments as a condition of incubation, 
and that would be a mistake because it presents a chicken-and-egg 
problem that would keep a lot of really useful projects out of 
OpenStack. But I don't think it's unreasonable to say that projects that 
haven't yet had the benefit of that feedback loop with operators and 
users must maximise their flexibility to take advantage of it when it 
appears without breaking the upgrade and API compatibility requirements 
that come with being a part of OpenStack. i.e. if your project is 
unproven, it had better be the simplest thing that could possibly work 
while still addressing the same problem; conversely if your project is 
breathtaking in scope and ambition (Zaqar is not in this boat, but... 
ahem... I could name you some examples) it had better come with some 
pretty strong evidence that it is really needed by users and accepted by 
operators.

cheers,
Zane.

> Flavio
>
>>
>> Excerpts from Joe Gordon's message of 2014-09-17 13:36:18 -0700:
>>> Hi All,
>>>
>>> My understanding of Zaqar is that it's like SQS. SQS uses distributed
>>> queues, which have a few unusual properties [0]:
>>> Message Order
>>>
>>> Amazon SQS makes a best effort to preserve order in messages, but due to
>>> the distributed nature of the queue, we cannot guarantee you will receive
>>> messages in the exact order you sent them. If your system requires that
>>> order be preserved, we recommend you place sequencing information in each
>>> message so you can reorder the messages upon receipt.
>>> At-Least-Once Delivery
>>>
>>> Amazon SQS stores copies of your messages on multiple servers for
>>> redundancy and high availability. On rare occasions, one of the servers
>>> storing a copy of a message might be unavailable when you receive or delete
>>> the message. If that occurs, the copy of the message will not be deleted on
>>> that unavailable server, and you might get that message copy again when you
>>> receive messages. Because of this, you must design your application to be
>>> idempotent (i.e., it must not be adversely affected if it processes the
>>> same message more than once).
>>> Message Sample
>>>
>>> The behavior of retrieving messages from the queue depends whether you are
>>> using short (standard) polling, the default behavior, or long polling. For
>>> more information about long polling, see Amazon SQS Long Polling
>>> <http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html>
>>> .
>>>
>>> With short polling, when you retrieve messages from the queue, Amazon SQS
>>> samples a subset of the servers (based on a weighted random distribution)
>>> and returns messages from just those servers. This means that a particular
>>> receive request might not return all your messages. Or, if you have a small
>>> number of messages in your queue (less than 1000), it means a particular
>>> request might not return any of your messages, whereas a subsequent request
>>> will. If you keep retrieving from your queues, Amazon SQS will sample all
>>> of the servers, and you will receive all of your messages.
>>>
>>> The following figure shows short polling behavior of messages being
>>> returned after one of your system components makes a receive request.
>>> Amazon SQS samples several of the servers (in gray) and returns the
>>> messages from those servers (Message A, C, D, and B). Message E is not
>>> returned to this particular request, but it would be returned to a
>>> subsequent request.
>>>
>>>
>>>
>>> Presumably SQS has these properties because it makes the system scalable,
>>> if so does Zaqar have the same properties (not just making these same
>>> guarantees in the API, but actually having these properties in the
>>> backends)? And if not, why? I looked on the wiki [1] for information on
>>> this, but couldn't find anything.
>>>
>>>
>>>
>>>
>>>
>>> [0]
>>> http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/DistributedQueues.html
>>> [1] https://wiki.openstack.org/wiki/Zaqar
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>