Open Stack

> On 08 Jul 2015, at 03:54, Zane Bitter <zbitter at redhat.com> wrote:
> 
> On 07/07/15 06:13, Renat Akhmerov wrote:
>> Just to clarify: what we’re looking for is how to implement “Work queue”
>> pattern described at [1] with oslo messaging. As Nikolay said, it
>> requires that a message to be acknowledged after it has been processed.
> 
> Yeah, so as you've discovered, oslo.messaging has no support for reliable asynchronous messaging. (Synchronous messaging in the form of 'call' can be somewhat reliable - in the sense that as long as the caller is still alive it can retry until it gets a reply.)

Yes, you managed to find the exactly right wording.

> In theory you could modify oslo.messaging to ack the message _after_ processing, but in practice I don't know how much that would help. IIRC Rabbit's acknowledgement mechanism still exhibits some flakiness when viewed from end-to-end, but even if that's wrong the very best you could say about it is that it's completely dependent on what settings are used in the RabbitMQ deployment - which are completely opaque to you. Since no project really 'owns' the semantics of Rabbit I don't believe there's any docs saying how it must be deployed, and if there is such a project then it's oslo.messaging and that acks messages before acting them so it's unlikely that it's docs call out the importance of reliable end-to-end delivery.

So yes. The thing I’m really really concerned about is the following.

Dealing with transport “sender-reicever” protocols is kind of a sience (at least for me) and patterns may vary from project to project just because of their specifics: load, QoS etc. etc. Quite often serious projects refuse to use even well known stable network/distributed frameworks just because abstractions provided by these frameworks hide required possibilities for them but they want to have everything under control. It’s important to be able to tune the system while doing performance testing and profiling which always reveals things you would never even expect. It’s the real world. So what I’m going to is that it’s kind of scary that I’m using RabbitMQ as a broker and I can’t, in fact, use all its power to do the best as I can. Right now we’re talking about reliable asynchronous processing (aka “Work queues”), tomorrow I may come to conclusion that if I had a chance to use whatever else mechanism of RabbitMQ that’s also hidden by the abstraction in the middle then I’d be able to provide even better solution.

Having that said, I think we could discuss how we could make oslo.messaging maybe more flexible in terms of using underlying transport specific features.

> The way I see it, there are two other options that would be preferable:
> 
> 1) Implement your own acks. Save the msg content to the DB, send a RabbitMQ message, delete the message from the DB once it's processed, have a periodic task to resend any messages that have timed out. Or
> 2) Zaqar


Renat Akhmerov
@ Mirantis Inc.