Open Stack

On 07/01/2014 03:52 PM, Ihar Hrachyshka wrote:
> On 01/07/14 15:55, Alexei Kornienko wrote:
>>> And in addition I've provided some links to existing
>>> implementation with places that IMHO cause bottlenecks. From my
>>> point of view that code is doing obviously stupid things (like
>>> closing/opening sockets for each message sent).
>
> That indeed sounds bad.
>
>>> That is enough for me to rewrite it even without additional
>>> proofs that it's wrong.
>
> [Full disclosure: I'm not as involved into oslo.messaging internals as
> you probably are, so I may speak out dumb things.]
>
> I wonder whether there are easier ways to fix that particular issue
> without rewriting everything from scratch. Like, provide a pool of
> connections and make send() functions use it instead of creating new
> connections (?)

I did a little investigation, using the test scripts from Alexei. 
Looking at the protocol trace with wireshark, you can certainly see lots 
of redundant stuff. However I did *not* see connections being opened and 
closed.

What I did see was the AMQP (0-9-1) channel being opened and closed, and 
the exchange being declared for every published message. Both these 
actions are synchronous. This clearly impacts throughput and, in my view 
more importantly, will limit scalability by giving the broker redundant 
work.

However, I think this could be avoided (at least to the extreme extent 
it occurs at present) by a simple patch. I posted this up to gerrit in 
case anyone is interested: https://review.openstack.org/#/c/104194/

With this in place the ingress improves significantly[1]. The bottleneck 
then appears to be the acking of messages. This means the queue depth 
tends to get higher than before, and the traffic appears more 'bursty'. 
Each message is acked individually, whereas some degree of batching 
might be possible which would likely improve the efficiency there a 
little bit.

Anyway, just an observation I thought it would be interesting to share.

--Gordon.

[1] From less than 400 msgs/sec to well over 1000 msgs/sec on my laptop