[openstack-dev] [oslo][messaging][pika] Pika driver development status

Dmitriy Ukhlov dukhlov at mirantis.com
Fri Apr 8 12:23:03 UTC 2016

Hi stackers!

In Mitaka new oslo.messaging driver for RabbitMQ is released (pika driver).
I would like to share information about current state of pika driver and plans for improvements
during next release cycle

Now we have driver implementation which:
1) passes all tempest tests
2) implements heartbeat mechanism properly (uses pika implementation instead of implementing by own, like kombu driver does)
3) implements fast detection of connection problem (tcp_user_timeout usage for sending, heartbeats for listening)
    and reconnection to next available connection node (works better with RabbitMQ cluster then kombu)
4) implements pure at-most-once message processing for RPC if retry=0 is set for message sending 
(kombu driver does not guarantee at-most-once processing even with retry=0 because uses acknowledges)
5) is about 50% faster then kombu (at least in my simple test with simulator.py - 1 rpc server process and 1 rpc client process, each client runs 5 threads):
	results for rabbit: 330.2 msg/sec
    	results for pika: 497.6 msg/sec
6) eats RabbitMQ a bit more then kombu (especially mandatory flag for rpc to fail fast if nobody listen target).
    Therefore in performance testing (17 rpc server processes and 17 rpc client processes, each client runs 5 threads),
    when RabbitMQ cluster is overloaded, pika driver works about 10% slower in rpc call. My results:
	results for rabbit: 3097 msg/sec
	results for pika: 2825 msg/sec
    but casts work faster about 17% then kombu because it is more lightweight and RabbitMQ is not so heavy loaded in my tests:
        results for rabbit: 5687 msg/sec
	results for pika: 6697 msg/sec
7) implements separately notifications and rpc messaging (using different exchanges, etc) which allows to use different configuration
    for different use cases (for example durable notification messaging and not durable rpc messaging)

Plans for future development:
1) Implement configurable message serialisation (json - current implementation, msgpack)
2) Implement configurable connection factory (pool of connection - current implementation, single connection)
3) Now it is impossible to perform rolling update from kombu to pika, we need to implements some solution for cross driver rolling update
4) Polishing, bug fixing, profiling etc., as usual

P.S. Thank everyone who have been helping to develop pika driver!


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160408/f0cba716/attachment.html>

More information about the OpenStack-dev mailing list