[openstack-dev] [oslo][messaging] Further improvements and refactoring

Ihar Hrachyshka ihrachys at redhat.com
Fri Jun 27 13:04:36 UTC 2014


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 26/06/14 22:38, Alexei Kornienko wrote:
> Hello Jay,
> 
> Benchmark for oslo.messaging is really simple: You create a client
> that sends messages infinitively and a server that processes them.
> After you can use rabbitmq management plugin to see average
> throughput in the queue. Simple example can be found here - 
> https://github.com/andreykurilin/profiler-for-oslo-messaging
> 
> I've mentioned some of this already in my previous mails but here
> it is again:
> 
>> "Huge" is descriptive but not quantitative :) Do you have any
>> numbers that pinpoint the amount of time that is being spent
>> reconstructing and declaring the queues, say, compared to the
>> time spent doing transmission?
> I don't have precise slowdown percentage for each issue. I've just 
> identified hotspots using cProfile and strace.
> 
>> This ten times number... is that an estimate or do you have hard 
>> numbers that show that? Just curious.
> Using a script that I've mentioned earlier I get average throughput
> on my PC ~700 cast calls per second. I've written a simple and
> stupid script that uses kombu directly (in a single threaded and
> synchronous way with single connection) It gave me throughput ~8000
> messages per second. Thats why I say that library should work at
> least 10 times faster.

It doesn't show that those major issues you've pointed out result in
such large message processing speed dropdown though. Maybe there are
other causes of slowdown we observe. Neither it shows that refactoring
of the code will actually help to boost the library significantly.

Though having some hard numbers from using kombu directly is still a
good thing. Is it possible that we introduce some performance tests
into oslo.messaging itself?

> 
> Regards, Alexei Kornienko
> 
> 
> On 06/26/2014 11:22 PM, Jay Pipes wrote:
>> Hey Alexei, thanks for sharing your findings. Comments inline.
>> 
>> On 06/26/2014 10:08 AM, Alexei Kornienko wrote:
>>> Hello,
>>> 
>>> Returning to performance issues of oslo.messaging. I've found 2
>>> biggest issue in existing implementation of rabbit (kombu) 
>>> driver:
>>> 
>>> 1) For almost every message sent/received a new object of 
>>> Consumer/Publisher class is created. And each object of this
>>> class tries to declare it's queue even if it's already
>>> declared. 
>>> https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L159
>>>
>>>
>>>
>>> 
This causes a huge slowdown.
>> 
>> "Huge" is descriptive but not quantitative :) Do you have any
>> numbers that pinpoint the amount of time that is being spent
>> reconstructing and declaring the queues, say, compared to the
>> time spent doing transmission?
>> 
>>> 2) with issue #1 is fixed (I've applied a small hack to fix it
>>> in my repo) the next big issue araise. For every rpc message
>>> received a reply is sent when processing is done (it seems that
>>> reply is sent even for "cast" calls which it really strange to
>>> me). Reply sent is using connection pool to "speed up" replies.
>>> Due to bad implementation of custom connection pool for every
>>> message sent underlying connection channel is closed and
>>> reopened: 
>>> https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L689
>>>
>>>
>>>
>>> 
Cause of this major issues oslo.messaging performance is at least 10
>>> times slower than it could be.
>> 
>> This ten times number... is that an estimate or do you have hard 
>> numbers that show that? Just curious.
>> 
>>> My opinion is that there is no simple and elegant fix for this
>>> issues in current implementation of oslo.messaging (most
>>> because of bad architecture of the library and processing
>>> flow). My proposal is that we should start working on new major
>>> release of messaging library with architectural issues fixed.
>>> This will allow us to avoid mentioned issues and provide much
>>> more performance and flexibility for end users. Main goal that
>>> we should achieve is separate rpc code from messaging code this
>>> will allow us to implement both parts in much simpler and 
>>> cleaner way and in the same time it would be much faster.
>> 
>> Perhaps actually a better starting point would be to create a 
>> benchmarking harness that will allow us to see some baseline 
>> throughput numbers that we can then compare to the iterative 
>> improvements you will push?
>> 
>> Best, -jay
>> 
>>> Please share your thoughts on this topic.
>>> 
>>> Regards, Alexei Kornienko
>>> 
>>> On 06/16/2014 02:47 PM, Gordon Sim wrote:
>>>> On 06/13/2014 02:06 PM, Ihar Hrachyshka wrote:
>>>>> On 10/06/14 15:40, Alexei Kornienko wrote:
>>>>>> On 06/10/2014 03:59 PM, Gordon Sim wrote:
>>>>>>> 
>>>>>>> I think there could be a lot of work required to
>>>>>>> significantly improve that driver, and I wonder if that
>>>>>>> would be better spent on e.g. the AMQP 1.0 driver which
>>>>>>> I believe will perform much better and will offer more
>>>>>>> choice in deployment.
>>>>>> 
>>>>>> I agree with you on this. However I'm not sure that we
>>>>>> can do such a decision. If we focus on amqp driver only
>>>>>> we should mention it explicitly and deprecate qpid driver
>>>>>> completely. There is no point in keeping driver that is
>>>>>> not really functional.
>>>>> 
>>>>> The driver is functional. It may be not that efficient as 
>>>>> alternatives, but that's not a valid reason to deprecate
>>>>> it.
>>>> 
>>>> The question in my view is what the plan is for ongoing
>>>> development.
>>>> 
>>>> Will the driver get better over time, or is it likely to
>>>> remain as is at best (or even deteriorate)?
>>>> 
>>>> Choice is good, but every choice adds to the maintenance
>>>> burden, in testing against regressions if nothing else.
>>>> 
>>>> I think an explicit decision about the future is beneficial,
>>>> whatever the decision may be.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________ OpenStack-dev
>>>> mailing list OpenStack-dev at lists.openstack.org 
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>>> 
_______________________________________________
>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org 
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>> 
_______________________________________________
>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>> 
> 
> _______________________________________________ OpenStack-dev
> mailing list OpenStack-dev at lists.openstack.org 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCgAGBQJTrWvkAAoJEC5aWaUY1u57ZmQH/iI+sZE90kt7joODJI3hW/Ov
uij7BuAD0gLHibrN7yhejpfz+a90RC4/PILmrxUxxkOxF+XW5c6uaBbJCf7330eX
nWzu3h3RSY6wZBuTgYMqdVpNQIx0ydnXZRUrX6ddfzjP17wDKLJP0e/PRJj/1JDC
QqU1z4Bmh6tJyq4V75aMFPNjy3uvWDjuIAE2XUMueQfmbJZAiyco2/3HlDZaNeF7
y/AdMW/NlwprVXH480pEt+7208aQ47GszrExgS6lqnji73d9eX+97CJwDLwQmxy5
YMThrTEyOquDcGJ/CEAViW1fIU0/I5KzlkFbCWxN1/975ZFA7qSvren9x1pir7E=
=5/I8
-----END PGP SIGNATURE-----



More information about the OpenStack-dev mailing list