[openstack-dev] [oslo][monasca] Can we uncap python-kafka ?
joe.keen at hpe.com
Thu Dec 1 16:26:43 UTC 2016
On 12/1/16, 3:11 AM, "Julien Danjou" <julien at danjou.info> wrote:
>On Thu, Dec 01 2016, Mehdi Abaakouk wrote:
>> I'm aware of all of that, oslo.messaging patch for the new version is
>> ready since 8 months now. And we are still blocked... capping libraries,
>> whatever the reason, is very annoying and just freezes people work.
>> From the API pov python-kafka haven't break anything, the API is still
>> here and documented (and deprecated). What monasca raises is performance
>> issue due to how their uses the library, and on absumption on how it
>> internally. Blocking all projects for that looks not fair to me.
>> As nadya said, now, we have users that that prefers using an unmerged
>> patch and the new lib instead of using upstream supported
>> version with the old lib. This is not an acceptable situation to me but
>> just my thought.
>> Where is the solution to allow oslo.messaging works blocked since 8
>> month to continue ?
>And if Monasca is using messaging, I wonder why they don't rely on
>oslo.messaging, which would also solve this entire problem in a snap.
Julien, I looked at the current oslo.messaging Kafka driver and
unfortunately I don¹t think it meets our needs. There are several major
pieces that are a problem.
The message context wrapped around every message is extra overhead we
There¹s no support for batch sends to Kafka.
There¹s no support for keyed producers.
The forced auto_commit on consumers isn¹t something we can tolerate.
There is no auto balance of multiple consumers in the same consumer group
on a topic.
The current Kafka driver we use in monasca-common has all these features.
Without them we can¹t meet the durability and performance levels we need.
What sort of performance levels is oslo.messaging tested at?
I¹ll look into testing the newest version of kafka-python and see if it
meets our needs. If it still isn¹t stable and performant enough what are
the available options?
More information about the OpenStack-dev