Open Stack

On 09/18/2014 07:19 PM, Joe Gordon wrote:
> 
> 
> On Thu, Sep 18, 2014 at 7:45 AM, Flavio Percoco <flavio at redhat.com
> <mailto:flavio at redhat.com>> wrote:
> 
>     On 09/18/2014 04:09 PM, Gordon Sim wrote:
>     > On 09/18/2014 12:31 PM, Flavio Percoco wrote:
>     >> On 09/17/2014 10:36 PM, Joe Gordon wrote:
>     >>> My understanding of Zaqar is that it's like SQS. SQS uses distributed
>     >>> queues, which have a few unusual properties [0]:
>     >>>
>     >>>
>     >>>      Message Order
>     >>>
>     >>> Amazon SQS makes a best effort to preserve order in messages, but due to
>     >>> the distributed nature of the queue, we cannot guarantee you will
>     >>> receive messages in the exact order you sent them. If your system
>     >>> requires that order be preserved, we recommend you place sequencing
>     >>> information in each message so you can reorder the messages upon
>     >>> receipt.
>     >>>
>     >>
>     >> Zaqar guarantees FIFO. To be more precise, it does that relying on the
>     >> storage backend ability to do so as well. Depending on the storage used,
>     >> guaranteeing FIFO may have some performance penalties.
>     >
>     > Would it be accurate to say that at present Zaqar does not use
>     > distributed queues, but holds all queue data in a storage mechanism of
>     > some form which may internally distribute that data among servers but
>     > provides Zaqar with a consistent data model of some form?
> 
>     I think this is accurate. The queue's distribution depends on the
>     storage ability to do so and deployers will be able to choose what
>     storage works best for them based on this as well. I'm not sure how
>     useful this separation is from a user perspective but I do see the
>     relevance when it comes to implementation details and deployments.
> 
> 
>     > [...]
>     >> As of now, Zaqar fully relies on the storage replication/clustering
>     >> capabilities to provide data consistency, availability and fault
>     >> tolerance.
>     >
>     > Is the replication synchronous or asynchronous with respect to client
>     > calls? E.g. will the response to a post of messages be returned only
>     > once the replication of those messages is confirmed? Likewise when
>     > deleting a message, is the response only returned when replicas of the
>     > message are deleted?
> 
>     It depends on the driver implementation and/or storage configuration.
>     For example, in the mongodb driver, we use the default write concern
>     called "acknowledged". This means that as soon as the message gets to
>     the master node (note it's not written on disk yet nor replicated) zaqar
>     will receive a confirmation and then send the response back to the
>     client. This is also configurable by the deployer by changing the
>     default write concern in the mongodb uri using
>     `?w=SOME_WRITE_CONCERN`[0].
> 
> 
> This means that by default Zaqar cannot guarantee a message will be
> delivered at all. A message can be acknowledged and then the 'master
> node' crashes and the message is lost. Zaqar's ability to guarantee
> delivery is limited by the reliability of a single node, while something
> like swift can only loose a piece of data if 3 machines crash at the
> same time.

Correct, as mentioned in my reply to Gordon, I also think `majority` is
a saner default for the write concern in this case.

I'm glad you mentioned Swift. We discussed a while back about having a
storage driver for it. I thought we had a blueprint for that but we
don't. Last time we discussed it, swift seemed to cover everything we
needed, IIRC. Anyway, just a thought.

Flavio

>     [0] http://docs.mongodb.org/manual/reference/connection-string/#uri.w
> 
>     >
>     >> However, as far as consuming messages is concerned, it can
>     >> guarantee once-and-only-once and/or at-least-once delivery depending on
>     >> the message pattern used to consume messages. Using pop or claims
>     >> guarantees the former whereas streaming messages out of Zaqar guarantees
>     >> the later.
>     >
>     > From what I can see, pop provides unreliable delivery (i.e. its similar
>     > to no-ack). If the delete call using pop fails while sending back the
>     > response, the messages are removed but didn't get to the client.
> 
>     Correct, pop works like no-ack. If you want to have pop+ack, it is
>     possible to claim just 1 message and then delete it.
> 
>     >
>     > What do you mean by 'streaming messages'?
> 
>     I'm sorry, that went out wrong. I had the "browsability" term in my head
>     and went with something even worse. By streaming messages I meant
>     polling messages without claiming them. In other words, at-least-once is
>     guaranteed by default, whereas once-and-only-once is guaranteed just if
>     claims are used.
> 
>     >
>     > [...]
>     >> Based on our short conversation on IRC last night, I understand you're
>     >> concerned that FIFO may result in performance issues. That's a valid
>     >> concern and I think the right answer is that it depends on the storage.
>     >> If the storage has a built-in FIFO guarantee then there's nothing Zaqar
>     >> needs to do there. In the other hand, if the storage does not have a
>     >> built-in support for FIFO, Zaqar will cover it in the driver
>     >> implementation. In the mongodb driver, each message has a marker that is
>     >> used to guarantee FIFO.
>     >
>     > That marker is a sequence number of some kind that is used to provide
>     > ordering to queries? Is it generated by the database itself?
> 
>     It's a sequence number to provide ordering to queries, correct.
>     Depending on the driver, it may be generated by Zaqar or the database.
>     In mongodb's case it's generated by Zaqar[0].
> 
>     [0]
>     https://github.com/openstack/zaqar/blob/master/zaqar/queues/storage/mongodb/queues.py#L103-L185
> 
>     --
>     @flaper87
>     Flavio Percoco
> 
>     _______________________________________________
>     OpenStack-dev mailing list
>     OpenStack-dev at lists.openstack.org
>     <mailto:OpenStack-dev at lists.openstack.org>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 


-- 
@flaper87
Flavio Percoco