Dear colleagues, does anyone have experience using RDMA-enabled CEPH with Openstack? How stable is it? Whether all Openstack components (Nova, Cinder) works with such specific configuration of Ceph? Any issues which can affect overall system reliability? Thank you. -- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison
Hi on 2019/8/28 15:58, Volodymyr Litovka wrote:
does anyone have experience using RDMA-enabled CEPH with Openstack? How stable is it? Whether all Openstack components (Nova, Cinder) works with such specific configuration of Ceph? Any issues which can affect overall system reliability?
I dont have experience on it in fact. But IMO ceph's data replicas and rebalancing don't rely on RDMA, will RDMA enabled machines improve the performance for cluster? regards.
Hi, starting Ceph v14, RDMA supported officially, so if you have NICs that support RDMA, you can try :) On 28.08.2019 11:14, Eliza wrote:
Hi
on 2019/8/28 15:58, Volodymyr Litovka wrote:
does anyone have experience using RDMA-enabled CEPH with Openstack? How stable is it? Whether all Openstack components (Nova, Cinder) works with such specific configuration of Ceph? Any issues which can affect overall system reliability?
I dont have experience on it in fact. But IMO ceph's data replicas and rebalancing don't rely on RDMA, will RDMA enabled machines improve the performance for cluster?
regards.
-- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison
Hi there -
On 28 Aug 2019, at 08:58, Volodymyr Litovka <doka.ua@gmx.com> wrote: does anyone have experience using RDMA-enabled CEPH with Openstack? How stable is it? Whether all Openstack components (Nova, Cinder) works with such specific configuration of Ceph? Any issues which can affect overall system reliability?
Last time I looked at this (with pre-release Nautilus, about 9 months ago), I had mixed results. There are four generally-available RDMA fabrics (Infiniband, RoCE, iWARP and OPA) and I had a go at all of them apart from iWARP. RoCE worked for me but IB and OPA were troublesome to get working. There’s some work contributed for iWARP support that introduces the RDMA connection manager (RDMACM), which I found also helped for IB and OPA. There is potential here but in performance terms, I didn’t manage a thorough benchmarking and didn’t see conclusive proof of advantage. Perhaps things have come on since I looked, but it wasn’t an obvious win at the time. I’d love to have another pop at it, but for lack of time… Cheers, Stig
Hi Stig, the main question is - whether you tested it with Openstack and all components works? Because regarding Ceph itself - we're using Mellanox ConnectX-4LX cards and found that Ceph works fine with RoCE in LAG configuration.
There is potential here
Yes, agree with you. That's why we're trying to be ready for future improvements :) I guess that since Ceph supports RDMA officially, Redhat (as owner of Ceph) will sell it and, thus, will improve support. Thank you. On 28.08.2019 11:42, Stig Telfer wrote:
Hi there -
On 28 Aug 2019, at 08:58, Volodymyr Litovka <doka.ua@gmx.com> wrote: does anyone have experience using RDMA-enabled CEPH with Openstack? How stable is it? Whether all Openstack components (Nova, Cinder) works with such specific configuration of Ceph? Any issues which can affect overall system reliability? Last time I looked at this (with pre-release Nautilus, about 9 months ago), I had mixed results.
There are four generally-available RDMA fabrics (Infiniband, RoCE, iWARP and OPA) and I had a go at all of them apart from iWARP.
RoCE worked for me but IB and OPA were troublesome to get working. There’s some work contributed for iWARP support that introduces the RDMA connection manager (RDMACM), which I found also helped for IB and OPA.
There is potential here but in performance terms, I didn’t manage a thorough benchmarking and didn’t see conclusive proof of advantage. Perhaps things have come on since I looked, but it wasn’t an obvious win at the time. I’d love to have another pop at it, but for lack of time…
Cheers, Stig
-- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison
Hi Volodymyr - When I was testing and benchmarking, it was outside of an OpenStack context. I’d be interested to know whether you saw a performance uplift. Are you using RDMA for both front-side and replication traffic? Best wishes, Stig
On 28 Aug 2019, at 10:27, Volodymyr Litovka <doka.ua@gmx.com> wrote:
Hi Stig,
the main question is - whether you tested it with Openstack and all components works? Because regarding Ceph itself - we're using Mellanox ConnectX-4LX cards and found that Ceph works fine with RoCE in LAG configuration.
There is potential here
Yes, agree with you. That's why we're trying to be ready for future improvements :) I guess that since Ceph supports RDMA officially, Redhat (as owner of Ceph) will sell it and, thus, will improve support.
Thank you.
On 28.08.2019 11:42, Stig Telfer wrote:
Hi there -
On 28 Aug 2019, at 08:58, Volodymyr Litovka <doka.ua@gmx.com> wrote: does anyone have experience using RDMA-enabled CEPH with Openstack? How stable is it? Whether all Openstack components (Nova, Cinder) works with such specific configuration of Ceph? Any issues which can affect overall system reliability? Last time I looked at this (with pre-release Nautilus, about 9 months ago), I had mixed results.
There are four generally-available RDMA fabrics (Infiniband, RoCE, iWARP and OPA) and I had a go at all of them apart from iWARP.
RoCE worked for me but IB and OPA were troublesome to get working. There’s some work contributed for iWARP support that introduces the RDMA connection manager (RDMACM), which I found also helped for IB and OPA.
There is potential here but in performance terms, I didn’t manage a thorough benchmarking and didn’t see conclusive proof of advantage. Perhaps things have come on since I looked, but it wasn’t an obvious win at the time. I’d love to have another pop at it, but for lack of time…
Cheers, Stig
-- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison
Hi Stig, as far as I understand, it's impossible to have RDMA on front-side, because QEMU uses librbd which talks to OSDs over TCP. Only replication is done over RDMA. Nevertheless I will keep you informed on the progress. On 28.08.2019 18:38, Stig Telfer wrote:
Hi Volodymyr -
When I was testing and benchmarking, it was outside of an OpenStack context.
I’d be interested to know whether you saw a performance uplift. Are you using RDMA for both front-side and replication traffic?
Best wishes, Stig
On 28 Aug 2019, at 10:27, Volodymyr Litovka <doka.ua@gmx.com> wrote:
Hi Stig,
the main question is - whether you tested it with Openstack and all components works? Because regarding Ceph itself - we're using Mellanox ConnectX-4LX cards and found that Ceph works fine with RoCE in LAG configuration.
There is potential here Yes, agree with you. That's why we're trying to be ready for future improvements :) I guess that since Ceph supports RDMA officially, Redhat (as owner of Ceph) will sell it and, thus, will improve support.
Thank you.
On 28.08.2019 11:42, Stig Telfer wrote:
Hi there -
On 28 Aug 2019, at 08:58, Volodymyr Litovka <doka.ua@gmx.com> wrote: does anyone have experience using RDMA-enabled CEPH with Openstack? How stable is it? Whether all Openstack components (Nova, Cinder) works with such specific configuration of Ceph? Any issues which can affect overall system reliability? Last time I looked at this (with pre-release Nautilus, about 9 months ago), I had mixed results.
There are four generally-available RDMA fabrics (Infiniband, RoCE, iWARP and OPA) and I had a go at all of them apart from iWARP.
RoCE worked for me but IB and OPA were troublesome to get working. There’s some work contributed for iWARP support that introduces the RDMA connection manager (RDMACM), which I found also helped for IB and OPA.
There is potential here but in performance terms, I didn’t manage a thorough benchmarking and didn’t see conclusive proof of advantage. Perhaps things have come on since I looked, but it wasn’t an obvious win at the time. I’d love to have another pop at it, but for lack of time…
Cheers, Stig
-- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison
-- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison
participants (3)
-
Eliza
-
Stig Telfer
-
Volodymyr Litovka