[openstack-dev] [oslo][monasca][requirements] Kafka 1.x for Oslo.Messaging and Monasca

Keen, Joe joe.keen at hpe.com
Wed Jun 29 02:47:29 UTC 2016


Tony,
  Psutil is the only library we’ve seen so far that was already in the
global requirements with a version we didn’t support.  Because of that we
don’t currently have it as a requirement for the Monasca Agent, we wanted
to use the upper constraints in our devstack gate jobs, and we just
install it externally for now.  Once that patch set lands we can add it
back into the requirements for the Monasca Agent.  We have several other
libraries we’re in the process of removing that aren’t strictly necessary
and that aren’t worth trying to add to global requirements right now.  We
currently have a review up to add our monasca-common repo to the
requirements [1].  Once that goes through we should be able to immediately
add reviews for our monasca-api and monasca-persister repos.


  I’m in the process of running some more Kafka tests and putting together
more details about the Kafka problems from the Monasca perspective.  The
short version is that we have a simple Kafka test we used to validate
version 0.9.5 that writes 100,000 messages to Kafka, at the average
message size that Monasca expects, and then records how long it takes to
consume them at a variety of fetch sizes from 50K to 2MB.  With 0.9.5,
depending on the fetch size, we could get throughput rates of 50K to 60K
per second.  With the 1.0.x series performance degraded down to 7K to 10K
per second and after running my test program repeatedly it eventually
consumed all the RAM in my VM and the kernel killed it.  I recently tested
version 1.2.2 and while the performance was better, a consistent ~40K per
second regardless of fetch size, that only worked for a single run of my
test program.  A subsequent run only managed ~12K per second and did not
complete the test.  It consumed ~7GB of RAM before running my VM out of
RAM.

Aside from those performance issues the 1.x library is a fundamental
change from the 0.9.x library.  The producers became inherently
asynchronous while Monasca requires synchronous writes.  We should be able
to configure the new producers to operate in a synchronous manner but we
have not been able to verify that.  The older synchronous producers have
been deprecated and based on bugs filed seem to have degraded reliability.
 The consumers attempt to balance themselves inherently now if you’re
using a Kafka version > 0.9 but we have not been able to verify that the
new consumer code functions with the consumer balancing methods that are
required with Kafka versions < 0.9.

We do want to upgrade once things stabilize, especially since we want to
take advantage of Kafka Streaming in the new 0.10 version of Kafka that
the 0.9.5 library doesn’t support, but because we’re still running into
performance and stability problems we haven’t been able to verify that the
new versions can support the Monasca use case.

[1] https://review.openstack.org/#/c/334625/


On 6/23/16, 9:37 PM, "Tony Breeds" <tony at bakeyournoodle.com> wrote:

>On Wed, Jun 22, 2016 at 05:34:27AM +0000, Keen, Joe wrote:
>> Davanum,
>>   We started work on getting Monasca into the global requirements with
>>two
>> reviews [1] [2] that add gate jobs and check requirements jobs for the
>> Monasca repositories.  Some repositories are being adapted to use
>>versions
>> of libraries that OpenStack currently accepts [3] and we¹re looking at
>>the
>> libraries we use that are not currently part of OpenStack and seeing if
>> they¹re worth trying to add to the global requirements.  We¹re hoping to
>> be able to start adding the global requirements reviews within a week or
>> two.
>> 
>> We definitely want to talk with the oslo.messaging team and explain the
>> ways we use Kafka and what effects the move to the 1.x versions of the
>> library has on us.  I¹ve attempted to contact the oslo.messaging team in
>> the oslo IRC channel to see if we can talk about this at a weekly
>>meeting
>> but I wasn¹t able to connect with anyone.  Would you prefer that
>> conversation happen on the mailing list here or could we add that topic
>>to
>> the next weekly meeting?
>> 
>> [1] https://review.openstack.org/#/c/316293/
>> [2] https://review.openstack.org/#/c/323567/
>
>These 2 are merged.
>
>> [3] https://review.openstack.org/#/c/323598/
>
>Taking a tangent here:
>
>In 2014[1] we added a cap to psutil because 2.x wasn't compatible with 1.x
>which is fine but 2 years latere we have 4.3.0 and because of the cap I'm
>guessing we've done very little to work towards 4.3.0
>
>I've added an item for the requirements team to look at what's involved in
>raising the minimum for psutil, but:
>    Requirement: psutil<2.0.0,>=1.1.1 (used by 41 projects)
>it wont happen soon.
>
>Is psutil the last of the "old" libraries you need to deal with?
>
>Getting back to the topic of kafka, what are the pain points involved in
>working with the 2.x API?  Clearly we're going to need to get monasca and
>oslo.messgaing on a compatible page RSN or we'll end up delaying things
>until
>Ocata :(
>
>Yours Tony.
>
>[1] https://review.openstack.org/#/c/81373/



More information about the OpenStack-dev mailing list