Open Stack

Excerpts from Mehdi Abaakouk's message of 2015-07-07 04:07:51 -0700:
> Hi,
> 
> The RPC API of oslo.messaging do it for you, you don't have to care 
> about acknowledgement (or anything else done by the driver because the 
> underlying used pattern depends of  it) .
> 

The original poster wasn't asking whether ack's are used, because that
much is clear, but rather why are acks happening before the message has
been processed. I would like to know why this is too. The message is
gone from the broker and if the worker dies, we are now reliant on the
caller to resend the message, except in most cases it doesn't, it just
logs the failure, which means OpenStack is less resilient to failure
because of this problem.

> For the "Working Queues" patterns, I guess what you need is to ensure 
> that the Target doesn't have the "server" attribute set and use "call" 
> or "cast" depending of your needs.
> 
> It works like this for rabbitmq:
>    * the message is acknowledged from the rabbitmq PoV when the worker 
> start the processing of the message
>    * when it finish it send a message back to the caller with the result 
> of the processing or the raised exception if that doesn't work
> 
> On the client side, when you use "call" is wait for the returns.
> 
> If you don't need to get the result or the exception occurred during the 
> message processing just use "cast", it doesn't wait for the return and 
> worker doesn't send it.
> 

At least call explodes in a fiery ball of python exception, because it is
RPC, and the caller is waiting for a response on the back channel. For
casts, the message is just gone forever if the worker dies while
processing it, and nobody will know.

To suggest this is intentional, or unintentional but good, is quite a
stretch, and I'm not at all convinced.

> When RPC is needed, acknowledgement after the message have been 
> processed is not enough reliable to ensure the message have been 
> processed correctly and can lead to stuck message on the queue.
> 

The ack isn't for the client. It is for the broker to say "you don't need
to keep trying to send this message anymore, I have done all I need to
ensure it reaches its destination." Of course for RPC you still need to
wait for a reply, and resend if you didn't get it. Except, all of the call
sites I checked _do not appear to resend_, they simply explode on timeout
waiting for reply. This is how calling code should work and I'm ok with
code in nova, cinder, et. al. being written this way, because I'd expect
my messaging layer to be at least somewhat reliable, but oslo.messaging
most certainly is not reliably delivering things with RabbitMQ.

It's understandable why this hasn't come up yet. Until now, we couldn't
even keep RabbitMQ _connected_, but heartbeats has helped with that. Now
we need to think about failure modes and how they can be handled better.
I think you'll find that once you try to make oslo.messaging handle the
retrying, that with the broker simply being ack'd all the time, you risk
duplicating RPC calls if you retry in a loop. The pattern is well
established in RabbitMQ that acks should happen _AFTER_ the message has
been consumed and thus should not be duplicated, not before.

> Le 2015-07-07 12:13, Renat Akhmerov a écrit :
> > Just to clarify: what we’re looking for is how to implement “Work
> > queue” pattern described at [1] with oslo messaging. As Nikolay said,
> > it requires that a message to be acknowledged after it has been
> > processed.
> > 
> > [1] http://www.rabbitmq.com/tutorials/tutorial-two-python.html
> > <http://www.rabbitmq.com/tutorials/tutorial-two-python.html>
> > 
> > Renat Akhmerov
> > @ Mirantis Inc.
> > 
> > 
> > 
> >> On 07 Jul 2015, at 15:58, Nikolay Makhotkin <nmakhotkin at mirantis.com> 
> >> wrote:
> >> 
> >> Hi,
> >> 
> >> I am using RabbitMQ as the backend and searched oslo.messaging for 
> >> message acknowledgement feature but I found only [1] what is wrong 
> >> using of acknowledgement since it acknowledges incoming message before 
> >> it has been processed (while it should be done only after processing 
> >> the message, otherwise we can lost given message if, say, the server 
> >> suddenly goes down).
> >> 
> >> So, my questions: does oslo.messaging indeed not support correct 
> >> acknowledgement feature? Or it is impossible to do for oslo.messaging 
> >> paradighm? Or is it somehow connected with that oslo.messaging has to 
> >> support a lot of transport types?
> >> 
> >> Can't it be implemented somehow in oslo.messaging?
> >> 
> >> [1] 
> >> https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/rpc/dispatcher.py#L135 
> >> <https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/rpc/dispatcher.py#L135>
> >> 
> >> 
> >> Best Regards,
> >> Nikolay
> >> @Mirantis Inc.
> >> __________________________________________________________________________
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: 
> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> > 
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: 
> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>