Open Stack

I have been working with Ken Giusti on an AMQP 1.0 messaging driver for 
olso.messaging. The code for this is now available for review at
https://review.openstack.org/#/c/75815/

As previously mentioned on this list, this uses the Apache Qpid Proton 
'protocol engine' library. This is not a standalone client library, it 
includes no IO or threading, and simply provides the encoding and 
protocol rules. This was chosen to give full flexibility in other 
aspects of the client, allowing it to be tailored to best suit the 
olso.messaging usecase. It does mean however that the driver has a 
little more code than might be expected had a full client library been used.

The code uses a new directory (package) layout for this driver, based on 
suggestions from Flavio Percoco (thanks Flavio!). There is a protocols 
directory under _drivers, with amqp being used for this one. This name 
was preferred to proton to make it clear that the intention is to speak 
clean, neutral AMQP 1.0 and to avoid tying the driver code to specific 
intermediaries.

With that new amqp package, there is a subdirectory called engine which 
contains some generic wrappers around the lower level proton APIs. It 
may be that at some point this layer, or something similar, is available 
either directly in proton or in some supplementary library at which 
point we would have the option of dropping some of the code from within 
oslo.messaging itself.

The driver.py module implements the defined driver API, mostly by making 
requests on the controller.py module where most of the protocol mapping 
logic lies. The io is driven by a separate thread and the event loop for 
this is defined in eventloop.py. The threads calling on the driver 
communicate with the io thread (which uses non-blocking io), using Queues.

So far the testing has largely been through ad hoc clients and servers 
for the oslo.messaging API. I have also used it with nova under 
devstack. My knowledge of openstack is still very low. I followed the 
very helpful advice offered by Russell Bryant here: 
https://ask.openstack.org/en/question/28/openstack-api-mocker-or-simulator/

However I am eager to learn more and any suggestions for things to carry 
out with real openstack service will be greatly appreciated. I also plan 
to work on some functional tests at the olso.messaging API level that 
could be used to test any of the drivers. This would allow the lack of 
covergae with the current qpid driver to be addressed as welll as 
providing more confidence with this new driver (and would I think be 
useful for any other driver implementer who lacks sufficient knowledge 
of the other openstack services, or simply as a way of catching more 
issues earlier).

The current driver available for review requires an intermediary of some 
form, whether a 'broker' that supports AMQP 1.0 or something slightly 
different such as the Qpid Dispatch Router previously mentioned on this 
list.

I have tested successfully with qpidd and qpid dispatch router. At 
present RabbitMQ does not support 'dynamic nodes' necessary for 
temporary reply queues in 1.0, but I have a workaround planned for that.

This email is already growing rather long, so I'll leave it at that for 
now but would be delighted to answer any questions or address nay 
feedback whether here or through the review request above, or the 
associated blueprint: 
https://blueprints.launchpad.net/oslo.messaging/+spec/amqp10-driver-implementation

I will communicate regarding enhancements and additions to the code.

--Gordon