Open Stack

On 12/2/16, 1:29 AM, "Mehdi Abaakouk" <sileht at sileht.net> wrote:

>On Fri, Dec 02, 2016 at 03:29:59PM +1100, Tony Breeds wrote:
>>On Thu, Dec 01, 2016 at 04:52:52PM +0000, Keen, Joe wrote:
>>
>>> Unfortunately there¹s nothing wrong on the Monasca side so far as we
>>>know.
>>>  We test new versions of the kafka-python library outside of Monasca
>>> before we bother to try integrating a new version.  Since 1.0 the
>>> kafka-python library has suffered from crashes and memory leaks severe
>>> enough that we¹ve never attempted using it in Monasca itself.  We
>>>reported
>>> the bugs we found to the kafka-python project but they were closed once
>>> they released a new version.
>>
>>So Opening bugs isn't working.  What about writing code?
>
>The bug https://github.com/dpkp/kafka-python/issues/55
>
>Reopening it would be the right solution here.
>
>I can't reproduce the segfault neither and I agree with dpkp, that looks
>like a
>ujson issue.


The bug I had was: https://github.com/dpkp/kafka-python/issues/551

In the case of that bug ujson was not an issue.  The behaviour remained
even using the standard json library.  The primary issue I found with it
was a memory leak over successive runs of the test script.  Eventually the
leak became so bad that the OOM killer killed the process which caused the
segfault I was seeing.  The last version I tested was 1.2.1 and it still
leaked badly.  I¹ll need to let the benchmark script run for a while and
make sure it¹s not still leaking.

>
>And my bench seems to confirm the perf issue have been solved:
>(but not in the pointed version...)
>
>$ pifpaf run kafka python kafka_test.py
>kafka-python version: 0.9.5
>...
>fetch size 179200 -> 45681.8728864 messages per second
>fetch size 204800 -> 47724.3810674 messages per second
>fetch size 230400 -> 47209.9841092 messages per second
>fetch size 256000 -> 48340.7719787 messages per second
>fetch size 281600 -> 49192.9896743 messages per second
>fetch size 307200 -> 50915.3291133 messages per second
>
>$ pifpaf run kafka python kafka_test.py
>kafka-python version: 1.0.2
>....
>fetch size 179200 -> 8546.77931323 messages per second
>fetch size 204800 -> 9213.30958314 messages per second
>fetch size 230400 -> 10316.668006 messages per second
>fetch size 256000 -> 11476.2285269 messages per second
>fetch size 281600 -> 12353.7254386 messages per second
>fetch size 307200 -> 13131.2367288 messages per second
>
>(1.1.1 and 1.2.5 have also the same issue)
>
>$ pifpaf run kafka python kafka_test.py
>kafka-python version: 1.3.1
>fetch size 179200 -> 44636.9371873 messages per second
>fetch size 204800 -> 44324.7085365 messages per second
>fetch size 230400 -> 45235.8283208 messages per second
>fetch size 256000 -> 45793.1044121 messages per second
>fetch size 281600 -> 44648.6357019 messages per second
>fetch size 307200 -> 44877.8445987 messages per second
>fetch size 332800 -> 47166.9176281 messages per second
>fetch size 358400 -> 47391.0057622 messages per second
>
>Looks like it works well now :)

It¹s good that the performance problem has been fixed.  The remaining
issues on the Monasca side are verifying that the batch send method we
were using in 0.9.5 still works with the new async behaviour, seeing if
our consumer auto balance still functions or converting to use the Kafka
internal auto balance in Kafka 0.10, and finding a way to do efficient
synchronous writes with the new async methods.