Open Stack

On Wed, Feb 20, 2013 at 01:19:58PM -0600, Ray Pekowski wrote:
> On Wed, Jan 30, 2013 at 9:48 AM, Eric Windisch <eric at cloudscaling.com>wrote:
> 
> > Furthermore, the code currently double-serializes all the content. Kombu
> > serializes data with JSON transparently, unless told otherwise, and the
> > same is done with ZeroMQ. At least with ZeroMQ, if the array is convertable
> > via bytes(), then we can send it without the double serialization, but the
> > envelope is currently a dict, so the double-serialization is still
> > required... I'm not so certain if we can avoid it for Kombu, but I do know
> > the default JSON serialization can be disabled.
> >
> > I was curious what the cost of this double serialization would be for the
> AMQP based Kombu RPC implementation (RabbitMQ).  I took an RPC throughput
> measurement with this set up:
> 
>    - All machines are 2 CPU VMs
>    - Three VMs:
>       - RPC load driver (caller)
>       - OpenStack simulated service (callee)
>       - 3 cluster RabbitMQ with mirroring enabled.
>    - 10 concurrent load generating threads achieving close to maximum
>    throughput
>    - The amqp_rpc_single_reply_queue enabled (fast RPCs)
>    - RPC calls (not casts)
> 
> In a prior study I used 9 load driver / simulated service pairs to drive
> higher throughput, but this single pair is good enough to get a rough idea.
> 
> Here are the results:
> 
> RPC envelope disabled:  170 RPCs/second
> RPC envelope enabled:  190 RPCs/second
> The performance cost of enabling RPC envelopes is about a 10% reduction RPC
> throughput.
> 
> The double serialization that Eric mentions is the reason for the slow
> down.  I think the only reason the double serialization was added was in
> preparation for signature signing.  If Grizzly goes out this way, then we
> will just have to take the 10% hit to AMQP RPC in Havana, when the envelope
> will be enabled with no option to disable it.  An alternative is to remove
> the JSON serialization from the current envelope code, but it might be too
> late for Grizzly.
> 
> I'm not really suggesting a change, more just mentioning as an FYI.

IIUC this '10% hit' is a micro-benchmark result - it isn't considering
the RPC overhead in the context of the other things in OpenStack that
consume CPU resources. If RPC only accounts for, say, 5% of CPU usage
across openstack services, then a 10% hit would only adding 0.5% overhead
to an OpenStack deployment as a whole.

So 10% sounds like alot of overhead, but my gut feeling is that it
may well be lost in the noise once you consider all the other things
going on in OpenStack services

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|