Open Stack

Thanks Joe for all these details, I can see that Monasca is still
not able to switch to the new lib for new very good reasons.

But according your comment on https://review.openstack.org/#/c/420579/ :

> I don't think that anyone currently using Monasca wants to accept either
> of those options so we need to find a way to maintain the current data
> guarantees while using the async behaviour of the new client library.
> That takes time and engineering effort to make that happen.  Is there
> anyone in the community willing to put in the effort to help build and
> test these new features at scale?

Nobody have plan to fix this issues soon.

And about from the same review:

> On another topic I'm curious what new features are you looking to get
> out of the new library.  Is there anything we can do to help you get the
> capabilities you want with the existing client?

I don't think asking other projects to use deprecated and unsupported
lib version in new code is good, it's just adding fresh technical debt.

So, I agree with gordc, perhaps you should stay with the old and
unsupported lib. And let other to use the supported one.


On Tue, Jan 17, 2017 at 11:58:25PM +0000, Keen, Joe wrote:
>Tony, I have some observations on the new client based on a short term
>test and a long running test.
>
>For short term use it uses 2x the memory compared to the older client.
>The logic that deals with receiving partial messages from Kafka was
>completely rewritten in the 1.x series and with logging enabled I see
>continual warnings about truncated messages.  I don’t lose any data
>because of this but I haven’t been able to verify if it’s doing more reads
>than necessary.  I don’t know that either of these problems are really a
>sticking point for Monasca but the increase in memory usage is potentially
>a problem.

>Long term testing showed some additional problems.  On a Kafka server that
>has been running for a couple weeks I can write data in but the
>kafka-python library is no longer able to read data from Kafka.  Clients
>written in other languages are able to read successfully.  Profiling of
>the python-kafka client shows that it’s spending all it’s time in a loop
>attempting to connect to Kafka:
>
>    27615    0.086    0.000    0.086    0.000 {method 'acquire’ of
>'thread.lock' objects}
>    43152    0.250    0.000    0.385    0.000 types.py:15(_unpack)
>    43153    0.135    0.000    0.135    0.000 {_struct.unpack}
>48040/47798    0.164    0.000    0.165    0.000 {len}
>    60351    0.201    0.000    0.201    0.000 {method 'read’ of
>'_io.BytesIO' objects}
>  7389962   23.985    0.000   23.985    0.000 {method 'keys' of ‘dict'
>objects}
>  7389999  104.931    0.000  395.654    0.000 conn.py:560(recv)
>  7389999   58.342    0.000  100.005    0.000
>conn.py:722(_requests_timed_out)
>  7389999   97.787    0.000  167.568    0.000 conn.py:588(_recv)
>  7390071   46.596    0.000   46.596    0.000 {method 'recv’ of
>'_socket.socket' objects}
>  7390145   23.151    0.000   23.151    0.000 conn.py:458(connected)
>  7390266   21.417    0.000   21.417    0.000 {method 'tell’ of
>'_io.BytesIO' objects}
>  7395664   41.695    0.000   41.695    0.000 {time.time}
>
>
>
>I also see additional problems with the use of the deprecated
>SimpleConsumer and SimpleProducer clients.  We really do need to
>investigate migrating to the new async only Producer objects while still
>maintaining the reliability guarantees that Monasca requires.
>
>
>On 12/13/16, 10:01 PM, "Tony Breeds" <tony at bakeyournoodle.com> wrote:
>
>>On Mon, Dec 05, 2016 at 04:03:13AM +0000, Keen, Joe wrote:
>>
>>> I don’t know, yet, that we can.  Unless we can find an answer to the
>>> questions I had above I’m not sure that this new library will be
>>> performant and durable enough for the use cases Monasca has.  I’m fairly
>>> confident that we can make it work but the performance issues with
>>> previous versions prevented us from even trying to integrate so it will
>>> take us some time.  If you need an answer more quickly than a week or
>>>so,
>>> and if anyone in the community is willing, I can walk them through the
>>> testing I’d expect to happen to validate the new library.
>>
>>Any updates Joe?  It's been 10 days and we're running close to Christamas
>>so
>>at this rate it'll be next year before we know if this is workable.
>>
>>Yours Tony.
>
>__________________________________________________________________________
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
Mehdi Abaakouk
mail: sileht at sileht.net
irc: sileht