[openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

Keen, Joe joe.keen at hpe.com
Thu Dec 1 16:26:43 UTC 2016

On 12/1/16, 3:11 AM, "Julien Danjou" <julien at danjou.info> wrote:

>On Thu, Dec 01 2016, Mehdi Abaakouk wrote:
>> I'm aware of all of that, oslo.messaging patch for the new version is
>> ready since 8 months now. And we are still blocked... capping libraries,
>> whatever the reason, is very annoying and just freezes people work.
>> From the API pov python-kafka haven't break anything, the API is still
>> here and documented (and deprecated). What monasca raises is performance
>> issue due to how their uses the library, and on absumption on how it
>> internally. Blocking all projects for that looks not fair to me.
>> As nadya said, now, we have users that that prefers using an unmerged
>> patch and the new lib instead of using upstream supported
>> version with the old lib. This is not an acceptable situation to me but
>> just my thought.
>> Where is the solution to allow oslo.messaging works blocked since 8
>> month to continue ?
>And if Monasca is using messaging, I wonder why they don't rely on
>oslo.messaging, which would also solve this entire problem in a snap.

Julien, I looked at the current oslo.messaging Kafka driver and
unfortunately I don¹t think it meets our needs.  There are several major
pieces that are a problem.

The message context wrapped around every message is extra overhead we
don¹t want.  
There¹s no support for batch sends to Kafka.
There¹s no support for keyed producers.
The forced auto_commit on consumers isn¹t something we can tolerate.
There is no auto balance of multiple consumers in the same consumer group
on a topic.

The current Kafka driver we use in monasca-common has all these features.
Without them we can¹t meet the durability and performance levels we need.

What sort of performance levels is oslo.messaging tested at?

I¹ll look into testing the newest version of kafka-python and see if it
meets our needs.  If it still isn¹t stable and performant enough what are
the available options?

More information about the OpenStack-dev mailing list