Open Stack

On Fri, Dec 02, 2016 at 03:29:59PM +1100, Tony Breeds wrote:
>On Thu, Dec 01, 2016 at 04:52:52PM +0000, Keen, Joe wrote:
>
>> Unfortunately there’s nothing wrong on the Monasca side so far as we know.
>>  We test new versions of the kafka-python library outside of Monasca
>> before we bother to try integrating a new version.  Since 1.0 the
>> kafka-python library has suffered from crashes and memory leaks severe
>> enough that we’ve never attempted using it in Monasca itself.  We reported
>> the bugs we found to the kafka-python project but they were closed once
>> they released a new version.
>
>So Opening bugs isn't working.  What about writing code?

The bug https://github.com/dpkp/kafka-python/issues/55

Reopening it would be the right solution here.

I can't reproduce the segfault neither and I agree with dpkp, that looks like a
ujson issue.

And my bench seems to confirm the perf issue have been solved:
(but not in the pointed version...)

$ pifpaf run kafka python kafka_test.py
kafka-python version: 0.9.5
...
fetch size 179200 -> 45681.8728864 messages per second
fetch size 204800 -> 47724.3810674 messages per second
fetch size 230400 -> 47209.9841092 messages per second
fetch size 256000 -> 48340.7719787 messages per second
fetch size 281600 -> 49192.9896743 messages per second
fetch size 307200 -> 50915.3291133 messages per second

$ pifpaf run kafka python kafka_test.py
kafka-python version: 1.0.2
....
fetch size 179200 -> 8546.77931323 messages per second
fetch size 204800 -> 9213.30958314 messages per second
fetch size 230400 -> 10316.668006 messages per second
fetch size 256000 -> 11476.2285269 messages per second
fetch size 281600 -> 12353.7254386 messages per second
fetch size 307200 -> 13131.2367288 messages per second

(1.1.1 and 1.2.5 have also the same issue)

$ pifpaf run kafka python kafka_test.py
kafka-python version: 1.3.1
fetch size 179200 -> 44636.9371873 messages per second
fetch size 204800 -> 44324.7085365 messages per second
fetch size 230400 -> 45235.8283208 messages per second
fetch size 256000 -> 45793.1044121 messages per second
fetch size 281600 -> 44648.6357019 messages per second
fetch size 307200 -> 44877.8445987 messages per second
fetch size 332800 -> 47166.9176281 messages per second
fetch size 358400 -> 47391.0057622 messages per second

Looks like it works well now :)


Just in case I have updated a bit the bench script to ensure it always works for me:

--- kafka_test.py.ori   2016-12-02 09:16:10.570677010 +0100
+++ kafka_test.py   2016-12-02 09:06:04.870370438 +0100
@@ -14,7 +14,7 @@
 import time
 import ujson
 
-KAFKA_URL = '92.168.10.6:9092'
+KAFKA_URL = '127.0.0.1:9092'
 KAFKA_GROUP = 'kafka_python_perf'
 KAFKA_TOPIC = 'raw-events'
 
@@ -24,6 +24,7 @@
 
 def write():
     k_client = KafkaClient(KAFKA_URL)
+    k_client.ensure_topic_exists(KAFKA_TOPIC)
     p = KeyedProducer(k_client,
                       async=False,
                       req_acks=KeyedProducer.ACK_AFTER_LOCAL_WRITE,


-- 
Mehdi Abaakouk
mail: sileht at sileht.net
irc: sileht