Open Stack

On Fri, Jan 25, 2013 at 3:07 PM, Russell Bryant <rbryant at redhat.com> wrote:

> Unfortunately I think there's a fundamental problem with any sort of
> solution that requires reissuing messages in a different format after a
> failure.  The messaging isn't connection oriented between the sender and
> receiver of a given message.  So, just because the last message failed
> doesn't mean the next one will.  The version implemented at the other
> end can change in between the original and re-sent message.
>

I think your point is arguable, but my proposal won't work for another
reason.  There is no fast way to detect a downlevel service in all cases.
I just looked at the ZMQ implementation and the new envelope causes the
dispatched service thread to stack dump with no response sent.  The RPC has
to time out, which isn't an acceptable way to handle a fallback retry.

I originally proposed this based upon what I know about the AMQP
implementation, which would return a failure reply in the case of the
'method' key missing in the request.

> And (optionally) perhaps even make a change in the checking of the
> > envelope to look for unexpected content in that envelope. For example,
> > if we decide in the future to add an implementation version or whatever,
> > that a version that supported the envelope, but not the new content
> > would fail the RPC."
>
> That's how the envelope works now.  It includes a version number.
> Locally we maintain a version number for the latest version that is
> supported.  An incoming message lists the minimum version that must be
> implemented for the message to be successfully processed.  If that check
> fails, an exception will be raised  If it gets updated to include
> additional information, the version number will get bumped.
>

This part of my proposal goes beyond the RPC version number and I think
still has merit.  My proposal is not only to check for known keys in the
envelope, but also to confirm that there are no unknown keys in the
envelope.  And if an unknown key is found, fail the RPC.  That way some
non-backward compatible change specific to amqp and not ZMQ could be
identified as a new entry in the envelope, for example as an amqp version
number.

Otherwise new non-backward compatible changes would need to go through two
releases.  The first for the option to be recognized but disabled and a
second for it to actually be enable, just as being done for the RPC
envelope.  With my proposal, we can skip the recognizing release, since by
definition an unknown entry in the envelope is recognized as a failure.

I'm not certain the exception that is raised today for version mismatch
results in a failed RPC.  That might need to be enhanced for my proposal to
work.

I think we should also try to avoid a new configuration option if at all
> possible.


Agreed, but it may not be possible.


> We just won't be turning that
> on for the sender-side until post-Grizzly.
>

I don't want to wait for post-Grizzly (or post H if it doesn't make it in
Grizzly)
for this RPC performance improvement.  Recall that this improvement takes a
cluster of 3 RabbitMQ servers with mirroring enabled from a maximum of 35
RPCs/sec to a maximum 800 RPCs/sec, a 23X improvement.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130126/36bc6d8b/attachment.html>