[openstack-dev] [oslo][messaging] Further improvements and refactoring

Flavio Percoco flavio at redhat.com
Tue Jun 10 12:06:49 UTC 2014


On 10/06/14 15:03 +0400, Dina Belova wrote:
>Hello, stackers!
>
>
>Oslo.messaging is future of how different OpenStack components communicate with
>each other, and really I’d love to start discussion about how we can make this
>library even better then it’s now and how can we refactor it make more
>production-ready.
>
>
>As we all remember, oslo.messaging was initially inspired to be created as a
>logical continuation of nova.rpc - as a separated library, with lots of
>transports supported, etc. That’s why oslo.messaging inherited not only
>advantages of now did the nova.rpc work (and it were lots of them), but also
>some architectural decisions that currently sometimes lead to the performance
>issues (we met some of them while Ceilometer performance testing [1] during the
>Icehouse).
>
>
>For instance, simple testing messaging server (with connection pool and
>eventlet) can process 700 messages per second. The same functionality
>implemented using plain kombu (without connection pool and eventlet)  driver is
>processing ten times more - 7000-8000 messages per second.
>
>
>So we have the following suggestions about how we may make this process better
>and quicker (and really I’d love to collect your feedback, folks):
>
>
>1) Currently we have main loop running in the Executor class, and I guess it’ll
>be much better to move it to the Server class, as it’ll make relationship
>between the classes easier and will leave Executor only one task - process the
>message and that’s it (in blocking or eventlet mode). Moreover, this will make
>further refactoring much easier.

To some extent, the executors are part of the server class since the
later is the one actually controlling them. If I understood your
proposal, the server class would implement the event loop, which means
we would have an EventletServer / BlockingServer, right?

If what I said is what you meant, then I disagree. Executors keep the
eventloop isolated from other parts of the library and this is really
important for us. One of the reason is to easily support multiple
python versions - by having different event loops.

Is my assumption correct? Could you elaborate more?

>
>2) Some of the drivers implementations (such as impl_rabbit and impl_qpid, for
>instance) are full of useless separated classes that in reality might be
>included to other ones. There are already some changes making the whole
>structure easier [2], and after the 1st issue will be solved Dispatcher and
>Listener also will be able to be refactored.

This was done on purpose. The idea was to focus on backwards
compatibility rather than cleaning up/improving the drivers. That
said, sounds like those drivers could user some clean up. However, I
think we should first extend the test suite a bit more before hacking
the existing drivers.


>
>3) If we’ll separate RPC functionality and messaging functionality it’ll make
>code base clean and easily reused.

What do you mean with this?

>
>4) Connection pool can be refactored to implement more efficient connection
>reusage.

Please, elaborate. What changes do you envision?

As Dims suggested, I think filing some specs for this (and keeping the
proposals separate) would help a lot in understanding what the exact
plan is.

Glad to know you're looking forward to help improving oslo.messaging.

Thanks,
Flavio

>Folks, are you ok with such a plan? Alexey Kornienko already started some of
>this work [2], but really we want to be sure that we chose the correct vector
>of development here.
>
>
>Thanks!
>
>
>[1] https://docs.google.com/document/d/
>1ARpKiYW2WN94JloG0prNcLjMeom-ySVhe8fvjXG_uRU/edit?usp=sharing
>
>[2] https://review.openstack.org/#/q/
>status:open+owner:akornienko+project:openstack/oslo.messaging,n,z
>
>
>Best regards,
>
>Dina Belova
>
>Software Engineer
>
>Mirantis Inc.
>

>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-- 
@flaper87
Flavio Percoco
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140610/3ce10e5d/attachment.pgp>


More information about the OpenStack-dev mailing list