Open Stack

Hi,

Cinder is lacking ZeroMQ messaging support in multibackend
configurations. This is due to the fact that we're abusing
``Target.server`` property by appending "@backend" suffix to the
hostname. This works just fine in RabbitMQ, as it routes messages using
queue names, but fails in ZeroMQ when 'server' property stops matching
hostnames.

There is a workaround for this in oslo.messaging [1], but it breaks
multibackend configrations, as backend A on a host can get message
addressed for the backend B on the same host.

We've seen tries [2], [3], [4] to fix it, but these implementations
were based on doing special operations when ZeroMQ is configured. This
invalidates concept of oslo.messaging being an abstraction layer.

My current idea to fix this [5] is to switch whole multibackend message
routing from being based on ``Target.server`` to ``Target.topic``.
Implementation is based on keeping the old RPC server that listens on
"cinder-volume" topic for fanout messages and creating a new RPC server
which listens on "cinder-volume.host at backend" topic and "host" server.
This is fine with rolling upgrades, as '$topic.$server' is how server
queue names are built in oslo.messaging, so we'll still listen on old
queues.

Please note that only the messaging layer is being changed. Internally
volumes will still be seen as existing on "host at backend" servers.

I've added an experimental gate-tempest-dsvm-zeromq-multibackend job,
which passes fine on [5], but fails on master, which proves that the
fix works.

Is anyone seeing dangers in this approach? If not - I'm asking for
reviews on [5].

Thanks,
Michal

[1] https://github.com/openstack/oslo.messaging/blob/7b5bec3133b6d74c4144fb8a33b9c9d2803e8858/oslo_messaging/_drivers/zmq_driver/zmq_address.py#L36-L39
[2] https://review.openstack.org/#/c/271848/
[3] https://review.openstack.org/#/c/277113/
[4] https://review.openstack.org/#/c/351862
[5] https://review.openstack.org/#/c/398452