Open Stack

On 12/1/16, 3:11 AM, "Julien Danjou" <julien at danjou.info> wrote:

>On Thu, Dec 01 2016, Mehdi Abaakouk wrote:
>
>> I'm aware of all of that, oslo.messaging patch for the new version is
>> ready since 8 months now. And we are still blocked... capping libraries,
>> whatever the reason, is very annoying and just freezes people work.
>>
>> From the API pov python-kafka haven't break anything, the API is still
>> here and documented (and deprecated). What monasca raises is performance
>> issue due to how their uses the library, and on absumption on how it
>>works
>> internally. Blocking all projects for that looks not fair to me.
>>
>> As nadya said, now, we have users that that prefers using an unmerged
>> patch and the new lib instead of using upstream supported
>> version with the old lib. This is not an acceptable situation to me but
>>that's
>> just my thought.
>>
>> Where is the solution to allow oslo.messaging works blocked since 8
>> month to continue ?
>
>+1
>
>And if Monasca is using messaging, I wonder why they don't rely on
>oslo.messaging, which would also solve this entire problem in a snap.


Julien, I looked at the current oslo.messaging Kafka driver and
unfortunately I don¹t think it meets our needs.  There are several major
pieces that are a problem.

The message context wrapped around every message is extra overhead we
don¹t want.  
There¹s no support for batch sends to Kafka.
There¹s no support for keyed producers.
The forced auto_commit on consumers isn¹t something we can tolerate.
There is no auto balance of multiple consumers in the same consumer group
on a topic.

The current Kafka driver we use in monasca-common has all these features.
Without them we can¹t meet the durability and performance levels we need.

What sort of performance levels is oslo.messaging tested at?

I¹ll look into testing the newest version of kafka-python and see if it
meets our needs.  If it still isn¹t stable and performant enough what are
the available options?