Open Stack

Hello,

Please find some answers inline.

Regards,
Alexei

On 06/27/2014 04:04 PM, Ihar Hrachyshka wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> On 26/06/14 22:38, Alexei Kornienko wrote:
>> Hello Jay,
>>
>> Benchmark for oslo.messaging is really simple: You create a client
>> that sends messages infinitively and a server that processes them.
>> After you can use rabbitmq management plugin to see average
>> throughput in the queue. Simple example can be found here -
>> https://github.com/andreykurilin/profiler-for-oslo-messaging
>>
>> I've mentioned some of this already in my previous mails but here
>> it is again:
>>
>>> "Huge" is descriptive but not quantitative :) Do you have any
>>> numbers that pinpoint the amount of time that is being spent
>>> reconstructing and declaring the queues, say, compared to the
>>> time spent doing transmission?
>> I don't have precise slowdown percentage for each issue. I've just
>> identified hotspots using cProfile and strace.
>>
>>> This ten times number... is that an estimate or do you have hard
>>> numbers that show that? Just curious.
>> Using a script that I've mentioned earlier I get average throughput
>> on my PC ~700 cast calls per second. I've written a simple and
>> stupid script that uses kombu directly (in a single threaded and
>> synchronous way with single connection) It gave me throughput ~8000
>> messages per second. Thats why I say that library should work at
>> least 10 times faster.
> It doesn't show that those major issues you've pointed out result in
> such large message processing speed dropdown though. Maybe there are
> other causes of slowdown we observe. Neither it shows that refactoring
> of the code will actually help to boost the library significantly.
It doesn't show that those major issues are the *only* reason for slowdown.
Bit it shows that those issues are biggest that are currently visible.
We'll get a major speed boost if we fix them (and possibly discover new 
issues the would prevent us of reaching full speed).
>
> Though having some hard numbers from using kombu directly is still a
> good thing. Is it possible that we introduce some performance tests
> into oslo.messaging itself?
Some performance tests may be introduced but they would be more like 
functional tests since they require setup of actual messaging server 
(rabbit, etc.).
What do you mean exactly by performance tests?
Just testing overall throughput with some basic scenarios or you mean 
finding hotspots in the code?

>
>> Regards, Alexei Kornienko
>>
>>
>> On 06/26/2014 11:22 PM, Jay Pipes wrote:
>>> Hey Alexei, thanks for sharing your findings. Comments inline.
>>>
>>> On 06/26/2014 10:08 AM, Alexei Kornienko wrote:
>>>> Hello,
>>>>
>>>> Returning to performance issues of oslo.messaging. I've found 2
>>>> biggest issue in existing implementation of rabbit (kombu)
>>>> driver:
>>>>
>>>> 1) For almost every message sent/received a new object of
>>>> Consumer/Publisher class is created. And each object of this
>>>> class tries to declare it's queue even if it's already
>>>> declared.
>>>> https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L159
>>>>
>>>>
>>>>
>>>>
> This causes a huge slowdown.
>>> "Huge" is descriptive but not quantitative :) Do you have any
>>> numbers that pinpoint the amount of time that is being spent
>>> reconstructing and declaring the queues, say, compared to the
>>> time spent doing transmission?
>>>
>>>> 2) with issue #1 is fixed (I've applied a small hack to fix it
>>>> in my repo) the next big issue araise. For every rpc message
>>>> received a reply is sent when processing is done (it seems that
>>>> reply is sent even for "cast" calls which it really strange to
>>>> me). Reply sent is using connection pool to "speed up" replies.
>>>> Due to bad implementation of custom connection pool for every
>>>> message sent underlying connection channel is closed and
>>>> reopened:
>>>> https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L689
>>>>
>>>>
>>>>
>>>>
> Cause of this major issues oslo.messaging performance is at least 10
>>>> times slower than it could be.
>>> This ten times number... is that an estimate or do you have hard
>>> numbers that show that? Just curious.
>>>
>>>> My opinion is that there is no simple and elegant fix for this
>>>> issues in current implementation of oslo.messaging (most
>>>> because of bad architecture of the library and processing
>>>> flow). My proposal is that we should start working on new major
>>>> release of messaging library with architectural issues fixed.
>>>> This will allow us to avoid mentioned issues and provide much
>>>> more performance and flexibility for end users. Main goal that
>>>> we should achieve is separate rpc code from messaging code this
>>>> will allow us to implement both parts in much simpler and
>>>> cleaner way and in the same time it would be much faster.
>>> Perhaps actually a better starting point would be to create a
>>> benchmarking harness that will allow us to see some baseline
>>> throughput numbers that we can then compare to the iterative
>>> improvements you will push?
>>>
>>> Best, -jay
>>>
>>>> Please share your thoughts on this topic.
>>>>
>>>> Regards, Alexei Kornienko
>>>>
>>>> On 06/16/2014 02:47 PM, Gordon Sim wrote:
>>>>> On 06/13/2014 02:06 PM, Ihar Hrachyshka wrote:
>>>>>> On 10/06/14 15:40, Alexei Kornienko wrote:
>>>>>>> On 06/10/2014 03:59 PM, Gordon Sim wrote:
>>>>>>>> I think there could be a lot of work required to
>>>>>>>> significantly improve that driver, and I wonder if that
>>>>>>>> would be better spent on e.g. the AMQP 1.0 driver which
>>>>>>>> I believe will perform much better and will offer more
>>>>>>>> choice in deployment.
>>>>>>> I agree with you on this. However I'm not sure that we
>>>>>>> can do such a decision. If we focus on amqp driver only
>>>>>>> we should mention it explicitly and deprecate qpid driver
>>>>>>> completely. There is no point in keeping driver that is
>>>>>>> not really functional.
>>>>>> The driver is functional. It may be not that efficient as
>>>>>> alternatives, but that's not a valid reason to deprecate
>>>>>> it.
>>>>> The question in my view is what the plan is for ongoing
>>>>> development.
>>>>>
>>>>> Will the driver get better over time, or is it likely to
>>>>> remain as is at best (or even deteriorate)?
>>>>>
>>>>> Choice is good, but every choice adds to the maintenance
>>>>> burden, in testing against regressions if nothing else.
>>>>>
>>>>> I think an explicit decision about the future is beneficial,
>>>>> whatever the decision may be.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________ OpenStack-dev
>>>>> mailing list OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
> _______________________________________________
>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
> _______________________________________________
>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> _______________________________________________ OpenStack-dev
>> mailing list OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBCgAGBQJTrWvkAAoJEC5aWaUY1u57ZmQH/iI+sZE90kt7joODJI3hW/Ov
> uij7BuAD0gLHibrN7yhejpfz+a90RC4/PILmrxUxxkOxF+XW5c6uaBbJCf7330eX
> nWzu3h3RSY6wZBuTgYMqdVpNQIx0ydnXZRUrX6ddfzjP17wDKLJP0e/PRJj/1JDC
> QqU1z4Bmh6tJyq4V75aMFPNjy3uvWDjuIAE2XUMueQfmbJZAiyco2/3HlDZaNeF7
> y/AdMW/NlwprVXH480pEt+7208aQ47GszrExgS6lqnji73d9eX+97CJwDLwQmxy5
> YMThrTEyOquDcGJ/CEAViW1fIU0/I5KzlkFbCWxN1/975ZFA7qSvren9x1pir7E=
> =5/I8
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev