Open Stack

On 11/06/14 18:01 +0000, Janczuk, Tomasz wrote:
>Thanks Flavio, some comments inline below.
>
>On 6/11/14, 5:15 AM, "Flavio Percoco" <flavio at redhat.com> wrote:
>
>>>
>>>  1.  Marconi exposes HTTP APIs that allow messages to be listed without
>>>consuming them. This API cannot be implemented on top of AMQP 0.9 which
>>>implements a strict queueing semantics.
>>
>>I believe this is quite an important endpoint for Marconi. It's not
>>about listing messages but getting batch of messages. Wether it is
>>through claims or not doesn't really matter. What matters is giving
>>the user the opportunity to get a set of messages, do some work and
>>decide what to do with those messages afterwards.
>
>The sticky point here is that this Marconi¹s endpoint allows messages to
>be obtained *without* consuming them in the traditional messaging system
>sense: the messages remain visible to other consumers. It could be argued
>that such semantics can be implemented on top of AMQP by first getting the
>messages and then immediately releasing them for consumption by others,
>before the Marconi call returns. However, even that is only possible for
>messages that are at the front of the queue - the "paging" mechanism using
>markers cannot be supported.

What matters is whether the listing functionality is useful or not.
Lets not think about it as "listing" or "paging" but about getting
batches of messages that are still available for others to process in
parallel. As mentioned in my previous email, AMQP has been a good way
to analyze the extra set of features Marconi exposes in the API but I
don't want to make the choice of usability based on whether
traditional messaging systems support it and how it could be
implemented there.

>>>  5.  Marconi message consumption API creates a ³claim ID² for a set of
>>>consumed messages, up to a limit. In the AMQP 0.9 model (as well as SQS
>>>and Azure Queues), ³claim ID² maps onto the concept of ³delivery tag²
>>>which has a 1-1 relationship with a message. Since there is no way to
>>>represent the 1-N mapping between claimID and messages in the AMQP 0.9
>>>model, it effectively restrict consumption of messages to one per
>>>claimID. This in turn prevents batch consumption benefits.
>>>
>>>  6.  Marconi message consumption acknowledgment requires both claimID
>>>and messageID to be provided. MessageID concept is missing in AMQP 0.9.
>>>In order to implement this API, assuming the artificial 1-1 restriction
>>>of claim-message mapping from #5 above, this API could be implemented by
>>>requiring that messageID === claimID. This is really a workaround.
>>>
>>
>>These 2 points represent quite a change in the way Marconi works and a
>>trade-off in terms of batch consumption (as you mentioned). I believe
>>we can have support for both things. For example, claimID+suffix where
>>suffix point to a specific claimed messages.
>>
>>I don't want to start an extended discussion about this here but lets
>>keep in mind that we may be able to support both. I personally think
>>Marconi's claim's are reasonable as they are, which means I currently
>>like them better than SQS's.
>
>What are the advantages of the Marconi model for claims over the SQS and
>Azure Queue model for acknowledgements?
>
>I think the SQS and Azure Queue model is both simpler and more flexible.
>But the key advantage is that it has been around for a while, has been
>proven to work, and people understand it.
>
>1. SQS and Azure require only one concept to acknowledge a message
>(receipt handle/pop receipt) as opposed to Marconi¹s two concepts (message
>ID + claim ID). SQS/Azure model is simpler.

TBH, I'm not exactly sure where you're going with this. I mean, the
model may look simpler but it's not necessarily better nor easier to
implement. Keeping both, messages and claims, separate in terms of IDs
and management is flexible and powerful enough, IMHO. But I'm probably
missing your point.

I don't believe requiring the messageID+ClaimID to delete a specific,
claimed, messages is hard.

>
>2. Similarly to Marconi, SQS and Azure allow individual claimed messages
>to be deleted. This is a wash.

Calling it a wash is neither helpful nor friendly. Why do you think it
is a wash?

Claiming a message does not delete the message, which means consumers
may want to delete it before the claim is released. Do you have a
better way to do it?

>
>3. SQS and Azure allow a batch of messages *up to a particular receipt
>handle/pop receit* to be deleted. This is more flexible than Marconi¹s
>mechanism or deleting all messages associated with a particular claim, and
>works very well for the most common scenario of in-order message delivery.

Pop semantic is on its way to the codebase. Limited claim deletes
sounds like an interesting thing, lets talk about it. Want to submit a
new spec?

>>>  7.  RabbitMQ message acknowledgment MUST be sent over the same AMQP
>>>channel instance on which the message was originally received. This
>>>requires that the two Marconi HTTP calls that receive and acknowledge a
>>>message are affinitized to the same Marconi backend. It either
>>>substantially complicates driver implementation (server-side reverse
>>>proxing of requests) or adds new requirements onto the Marconi
>>>deployment (server affinity through load balancing).
>>>
>>
>>Nothing to do here. I guess a combination with a persistent-transport
>>protocol would help but we don't want to make the store driver depend
>>on the transport.
>
>The bigger issue here is that the storage layer abstraction has been
>design with a focus on a set of atomic, *independent* operations, with no
>way to correlate them or share any state between them. This works well for
>transport layers that are operation-centered, like HTTP. This abstraction
>is likely to be insufficient when one wants to use a connection-oriented
>transport (e.g. WebSocket, AMQP, MQTT), or a storage driver based on a
>connection-oriented protocol (e.g. AMQP). Without explicit modeling of a
>state that spans several atomic calls to the storage driver, in particular
>around the lifetime of the connection, current storage abstraction cannot
>satisfy these configurations well.

This is the current status, True. However, it was not always like
that. We decided to go this way until we actually needed to support
other transports (YAGNI). The time to revisit this is, perhaps,
coming.

That said, I agree that some things should change in the storage to
support other transports. But this is a different discussion from the
changes in the API to support other storage drivers.

Flavio

-- 
@flaper87
Flavio Percoco
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140613/1e1bd750/attachment.pgp>