[openstack-dev] [marconi] RabbitMQ (AMQP 0.9) driver for Marconi

Janczuk, Tomasz tomasz.janczuk at hp.com
Fri Jun 13 17:18:45 UTC 2014


Thanks Flavio, inline.

On 6/13/14, 1:37 AM, "Flavio Percoco" <flavio at redhat.com> wrote:

>On 11/06/14 18:01 +0000, Janczuk, Tomasz wrote:
>>Thanks Flavio, some comments inline below.
>>
>>On 6/11/14, 5:15 AM, "Flavio Percoco" <flavio at redhat.com> wrote:
>>
>>>>
>>>>  1.  Marconi exposes HTTP APIs that allow messages to be listed
>>>>without
>>>>consuming them. This API cannot be implemented on top of AMQP 0.9 which
>>>>implements a strict queueing semantics.
>>>
>>>I believe this is quite an important endpoint for Marconi. It's not
>>>about listing messages but getting batch of messages. Wether it is
>>>through claims or not doesn't really matter. What matters is giving
>>>the user the opportunity to get a set of messages, do some work and
>>>decide what to do with those messages afterwards.
>>
>>The sticky point here is that this Marconi¹s endpoint allows messages to
>>be obtained *without* consuming them in the traditional messaging system
>>sense: the messages remain visible to other consumers. It could be argued
>>that such semantics can be implemented on top of AMQP by first getting
>>the
>>messages and then immediately releasing them for consumption by others,
>>before the Marconi call returns. However, even that is only possible for
>>messages that are at the front of the queue - the "paging" mechanism
>>using
>>markers cannot be supported.
>
>What matters is whether the listing functionality is useful or not.
>Lets not think about it as "listing" or "paging" but about getting
>batches of messages that are still available for others to process in
>parallel. As mentioned in my previous email, AMQP has been a good way
>to analyze the extra set of features Marconi exposes in the API but I
>don't want to make the choice of usability based on whether
>traditional messaging systems support it and how it could be
>implemented there.

This functionality is very useful in a number of scenarios. It has
traditionally been the domain of database systems - flexible access to
data is what DBs excel in (select top 1000 * from X order by create_date).
Vast majority of existing messaging systems has a much more constrained
and prescriptive way of accessing data than a database. Why does this
functionality need to be part of Marconi? What are the benefits of listing
messages in Marconi that cannot be realized with a plain database?

In other words, if I need to access my data in that way, why would I use
Marconi rather than a DB?

>
>>>>  5.  Marconi message consumption API creates a ³claim ID² for a set of
>>>>consumed messages, up to a limit. In the AMQP 0.9 model (as well as SQS
>>>>and Azure Queues), ³claim ID² maps onto the concept of ³delivery tag²
>>>>which has a 1-1 relationship with a message. Since there is no way to
>>>>represent the 1-N mapping between claimID and messages in the AMQP 0.9
>>>>model, it effectively restrict consumption of messages to one per
>>>>claimID. This in turn prevents batch consumption benefits.
>>>>
>>>>  6.  Marconi message consumption acknowledgment requires both claimID
>>>>and messageID to be provided. MessageID concept is missing in AMQP 0.9.
>>>>In order to implement this API, assuming the artificial 1-1 restriction
>>>>of claim-message mapping from #5 above, this API could be implemented
>>>>by
>>>>requiring that messageID === claimID. This is really a workaround.
>>>>
>>>
>>>These 2 points represent quite a change in the way Marconi works and a
>>>trade-off in terms of batch consumption (as you mentioned). I believe
>>>we can have support for both things. For example, claimID+suffix where
>>>suffix point to a specific claimed messages.
>>>
>>>I don't want to start an extended discussion about this here but lets
>>>keep in mind that we may be able to support both. I personally think
>>>Marconi's claim's are reasonable as they are, which means I currently
>>>like them better than SQS's.
>>
>>What are the advantages of the Marconi model for claims over the SQS and
>>Azure Queue model for acknowledgements?
>>
>>I think the SQS and Azure Queue model is both simpler and more flexible.
>>But the key advantage is that it has been around for a while, has been
>>proven to work, and people understand it.
>>
>>1. SQS and Azure require only one concept to acknowledge a message
>>(receipt handle/pop receipt) as opposed to Marconi¹s two concepts
>>(message
>>ID + claim ID). SQS/Azure model is simpler.
>
>TBH, I'm not exactly sure where you're going with this. I mean, the
>model may look simpler but it's not necessarily better nor easier to
>implement. Keeping both, messages and claims, separate in terms of IDs
>and management is flexible and powerful enough, IMHO. But I'm probably
>missing your point.
>
>I don't believe requiring the messageID+ClaimID to delete a specific,
>claimed, messages is hard.

It may not be hard. It is just more complex than it needs to be to
accomplish the same task.

>
>>
>>2. Similarly to Marconi, SQS and Azure allow individual claimed messages
>>to be deleted. This is a wash.
>
>Calling it a wash is neither helpful nor friendly. Why do you think it
>is a wash?
>
>Claiming a message does not delete the message, which means consumers
>may want to delete it before the claim is released. Do you have a
>better way to do it?

By "wash" I mean neither SQS/Azure or Marconi offers something the other
does not related to this aspect. I don't see a better way of doing it,
this is pretty standard.


>
>>
>>3. SQS and Azure allow a batch of messages *up to a particular receipt
>>handle/pop receit* to be deleted. This is more flexible than Marconi¹s
>>mechanism or deleting all messages associated with a particular claim,
>>and
>>works very well for the most common scenario of in-order message
>>delivery.
>
>Pop semantic is on its way to the codebase. Limited claim deletes
>sounds like an interesting thing, lets talk about it. Want to submit a
>new spec?

I will be happy to propose a spec, but I'd rather take a more
comprehensive approach than addressing this single issue. We have
discussed a number of problems both on this thread and outside of it.
Perhaps all this learning should come together as a V2 of the HTTP API.

>
>>>>  7.  RabbitMQ message acknowledgment MUST be sent over the same AMQP
>>>>channel instance on which the message was originally received. This
>>>>requires that the two Marconi HTTP calls that receive and acknowledge a
>>>>message are affinitized to the same Marconi backend. It either
>>>>substantially complicates driver implementation (server-side reverse
>>>>proxing of requests) or adds new requirements onto the Marconi
>>>>deployment (server affinity through load balancing).
>>>>
>>>
>>>Nothing to do here. I guess a combination with a persistent-transport
>>>protocol would help but we don't want to make the store driver depend
>>>on the transport.
>>
>>The bigger issue here is that the storage layer abstraction has been
>>design with a focus on a set of atomic, *independent* operations, with no
>>way to correlate them or share any state between them. This works well
>>for
>>transport layers that are operation-centered, like HTTP. This abstraction
>>is likely to be insufficient when one wants to use a connection-oriented
>>transport (e.g. WebSocket, AMQP, MQTT), or a storage driver based on a
>>connection-oriented protocol (e.g. AMQP). Without explicit modeling of a
>>state that spans several atomic calls to the storage driver, in
>>particular
>>around the lifetime of the connection, current storage abstraction cannot
>>satisfy these configurations well.
>
>This is the current status, True. However, it was not always like
>that. We decided to go this way until we actually needed to support
>other transports (YAGNI). The time to revisit this is, perhaps,
>coming.
>
>That said, I agree that some things should change in the storage to
>support other transports. But this is a different discussion from the
>changes in the API to support other storage drivers.

Yes, this is a separate discussion. I will start a new thread to talk
about this. 

>
>Flavio
>
>-- 
>@flaper87
>Flavio Percoco
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



More information about the OpenStack-dev mailing list