Hi there -
On 28 Aug 2019, at 08:58, Volodymyr Litovka <doka.ua@gmx.com> wrote: does anyone have experience using RDMA-enabled CEPH with Openstack? How stable is it? Whether all Openstack components (Nova, Cinder) works with such specific configuration of Ceph? Any issues which can affect overall system reliability?
Last time I looked at this (with pre-release Nautilus, about 9 months ago), I had mixed results. There are four generally-available RDMA fabrics (Infiniband, RoCE, iWARP and OPA) and I had a go at all of them apart from iWARP. RoCE worked for me but IB and OPA were troublesome to get working. There’s some work contributed for iWARP support that introduces the RDMA connection manager (RDMACM), which I found also helped for IB and OPA. There is potential here but in performance terms, I didn’t manage a thorough benchmarking and didn’t see conclusive proof of advantage. Perhaps things have come on since I looked, but it wasn’t an obvious win at the time. I’d love to have another pop at it, but for lack of time… Cheers, Stig