Hi Stig, as far as I understand, it's impossible to have RDMA on front-side, because QEMU uses librbd which talks to OSDs over TCP. Only replication is done over RDMA. Nevertheless I will keep you informed on the progress. On 28.08.2019 18:38, Stig Telfer wrote:
Hi Volodymyr -
When I was testing and benchmarking, it was outside of an OpenStack context.
I’d be interested to know whether you saw a performance uplift. Are you using RDMA for both front-side and replication traffic?
Best wishes, Stig
On 28 Aug 2019, at 10:27, Volodymyr Litovka <doka.ua@gmx.com> wrote:
Hi Stig,
the main question is - whether you tested it with Openstack and all components works? Because regarding Ceph itself - we're using Mellanox ConnectX-4LX cards and found that Ceph works fine with RoCE in LAG configuration.
There is potential here Yes, agree with you. That's why we're trying to be ready for future improvements :) I guess that since Ceph supports RDMA officially, Redhat (as owner of Ceph) will sell it and, thus, will improve support.
Thank you.
On 28.08.2019 11:42, Stig Telfer wrote:
Hi there -
On 28 Aug 2019, at 08:58, Volodymyr Litovka <doka.ua@gmx.com> wrote: does anyone have experience using RDMA-enabled CEPH with Openstack? How stable is it? Whether all Openstack components (Nova, Cinder) works with such specific configuration of Ceph? Any issues which can affect overall system reliability? Last time I looked at this (with pre-release Nautilus, about 9 months ago), I had mixed results.
There are four generally-available RDMA fabrics (Infiniband, RoCE, iWARP and OPA) and I had a go at all of them apart from iWARP.
RoCE worked for me but IB and OPA were troublesome to get working. There’s some work contributed for iWARP support that introduces the RDMA connection manager (RDMACM), which I found also helped for IB and OPA.
There is potential here but in performance terms, I didn’t manage a thorough benchmarking and didn’t see conclusive proof of advantage. Perhaps things have come on since I looked, but it wasn’t an obvious win at the time. I’d love to have another pop at it, but for lack of time…
Cheers, Stig
-- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison
-- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison