Open Stack

Hi Stig,

as far as I understand, it's impossible to have RDMA on front-side,
because QEMU uses librbd which talks to OSDs over TCP. Only replication
is done over RDMA.

Nevertheless I will keep you informed on the progress.

On 28.08.2019 18:38, Stig Telfer wrote:
> Hi Volodymyr -
>
> When I was testing and benchmarking, it was outside of an OpenStack context.
>
> I’d be interested to know whether you saw a performance uplift.  Are you using RDMA for both front-side and replication traffic?
>
> Best wishes,
> Stig
>
>
>> On 28 Aug 2019, at 10:27, Volodymyr Litovka <doka.ua at gmx.com> wrote:
>>
>> Hi Stig,
>>
>> the main question is - whether you tested it with Openstack and all
>> components works? Because regarding Ceph itself - we're using Mellanox
>> ConnectX-4LX cards and found that Ceph works fine with RoCE in LAG
>> configuration.
>>
>>> There is potential here
>> Yes, agree with you. That's why we're trying to be ready for future
>> improvements :) I guess that since Ceph supports RDMA officially, Redhat
>> (as owner of Ceph) will sell it and, thus, will improve support.
>>
>> Thank you.
>>
>> On 28.08.2019 11:42, Stig Telfer wrote:
>>> Hi there -
>>>
>>>> On 28 Aug 2019, at 08:58, Volodymyr Litovka <doka.ua at gmx.com> wrote:
>>>> does anyone have experience using RDMA-enabled CEPH with Openstack? How
>>>> stable is it? Whether all Openstack components (Nova, Cinder) works with
>>>> such specific configuration of Ceph? Any issues which can affect overall
>>>> system reliability?
>>> Last time I looked at this (with pre-release Nautilus, about 9 months ago), I had mixed results.
>>>
>>> There are four generally-available RDMA fabrics (Infiniband, RoCE, iWARP and OPA) and I had a go at all of them apart from iWARP.
>>>
>>> RoCE worked for me but IB and OPA were troublesome to get working.  There’s some work contributed for iWARP support that introduces the RDMA connection manager (RDMACM), which I found also helped for IB and OPA.
>>>
>>> There is potential here but in performance terms, I didn’t manage a thorough benchmarking and didn’t see conclusive proof of advantage.  Perhaps things have come on since I looked, but it wasn’t an obvious win at the time.  I’d love to have another pop at it, but for lack of time…
>>>
>>> Cheers,
>>> Stig
>>>
>>>
>> --
>> Volodymyr Litovka
>>   "Vision without Execution is Hallucination." -- Thomas Edison
>>
>>
>

--
Volodymyr Litovka
   "Vision without Execution is Hallucination." -- Thomas Edison