Open Stack

Tony, I have some observations on the new client based on a short term
test and a long running test.

For short term use it uses 2x the memory compared to the older client.
The logic that deals with receiving partial messages from Kafka was
completely rewritten in the 1.x series and with logging enabled I see
continual warnings about truncated messages.  I don’t lose any data
because of this but I haven’t been able to verify if it’s doing more reads
than necessary.  I don’t know that either of these problems are really a
sticking point for Monasca but the increase in memory usage is potentially
a problem.

Long term testing showed some additional problems.  On a Kafka server that
has been running for a couple weeks I can write data in but the
kafka-python library is no longer able to read data from Kafka.  Clients
written in other languages are able to read successfully.  Profiling of
the python-kafka client shows that it’s spending all it’s time in a loop
attempting to connect to Kafka:

    27615    0.086    0.000    0.086    0.000 {method 'acquire’ of
'thread.lock' objects}
    43152    0.250    0.000    0.385    0.000 types.py:15(_unpack)
    43153    0.135    0.000    0.135    0.000 {_struct.unpack}
48040/47798    0.164    0.000    0.165    0.000 {len}
    60351    0.201    0.000    0.201    0.000 {method 'read’ of
'_io.BytesIO' objects}
  7389962   23.985    0.000   23.985    0.000 {method 'keys' of ‘dict'
objects}
  7389999  104.931    0.000  395.654    0.000 conn.py:560(recv)
  7389999   58.342    0.000  100.005    0.000
conn.py:722(_requests_timed_out)
  7389999   97.787    0.000  167.568    0.000 conn.py:588(_recv)
  7390071   46.596    0.000   46.596    0.000 {method 'recv’ of
'_socket.socket' objects}
  7390145   23.151    0.000   23.151    0.000 conn.py:458(connected)
  7390266   21.417    0.000   21.417    0.000 {method 'tell’ of
'_io.BytesIO' objects}
  7395664   41.695    0.000   41.695    0.000 {time.time}



I also see additional problems with the use of the deprecated
SimpleConsumer and SimpleProducer clients.  We really do need to
investigate migrating to the new async only Producer objects while still
maintaining the reliability guarantees that Monasca requires.


On 12/13/16, 10:01 PM, "Tony Breeds" <tony at bakeyournoodle.com> wrote:

>On Mon, Dec 05, 2016 at 04:03:13AM +0000, Keen, Joe wrote:
>
>> I don’t know, yet, that we can.  Unless we can find an answer to the
>> questions I had above I’m not sure that this new library will be
>> performant and durable enough for the use cases Monasca has.  I’m fairly
>> confident that we can make it work but the performance issues with
>> previous versions prevented us from even trying to integrate so it will
>> take us some time.  If you need an answer more quickly than a week or
>>so,
>> and if anyone in the community is willing, I can walk them through the
>> testing I’d expect to happen to validate the new library.
>
>Any updates Joe?  It's been 10 days and we're running close to Christamas
>so
>at this rate it'll be next year before we know if this is workable.
>
>Yours Tony.