[cinder][nova] NVMe over RMDA
Hi all, does anyone know if NVMe over RDMA is supported to attach it with Cinder/Nova to a QemuKVM Hypervisor? In addition to our ceph, we would also like to connect a block storage and in order to be able to retrieve the full performance, this must be done with the above-mentioned connection. Thanks in advance. Holger
On 17/02/2025 12:58, holger.bielefeld@uponu.com wrote:
Hi all,
does anyone know if NVMe over RDMA is supported to attach it with Cinder/Nova to a QemuKVM Hypervisor?
technically yes. it depend on if you cinder driver supports nvme over rdma as well. ceph obviously does not as it use RBD to export the block storage. the mounting of the NVMEoF devicve is handled by os-brick so form a nova/qemu point of view its a host mounted block device. as such there is no difference between that and iscsi because qemu is not directly connecting to the storage backend.
In addition to our ceph, we would also like to connect a block storage and in order to be able to retrieve the full performance, this must be done with the above-mentioned connection.
Thanks in advance.
Holger
Hi Holger, Looking at the cinder support matrix[1], only Pure storage has declared support for RoCE (I think they mean ROCEv2 only as RoCEv1 is rarely used) and os-brick does support RoCEv2. It should technically "work" with Infiniband hardware, as the main protocol is RDMA either over infiniband or ethernet cards but I have no experience trying it so better to ask the vendor i.e. Pure about it. Probably @Simon Dodsley <simon@purestorage.com> can provide more information on it. [1] https://docs.openstack.org/cinder/latest/reference/support-matrix.html Thanks Rajat Dhasmana On Mon, Feb 17, 2025 at 7:09 PM Sean Mooney <smooney@redhat.com> wrote:
On 17/02/2025 12:58, holger.bielefeld@uponu.com wrote:
Hi all,
does anyone know if NVMe over RDMA is supported to attach it with Cinder/Nova to a QemuKVM Hypervisor?
technically yes.
it depend on if you cinder driver supports nvme over rdma as well.
ceph obviously does not as it use RBD to export the block storage.
the mounting of the NVMEoF devicve is handled by os-brick so form a nova/qemu point of view its a host mounted block device. as such there is no difference between that and iscsi because qemu is not directly connecting to the storage backend.
In addition to our ceph, we would also like to connect a block storage and in order to be able to retrieve the full performance, this must be done with the above-mentioned connection.
Thanks in advance.
Holger
Hi Yes, Pure does support NVMe-RoCE v2 for Cinder using our FlashArray product. It also supports NVMe-TCP. os-brick has been optimized to deal with NVMe connectivity with the 2023.2 release having the most comprehensive patches. If you want more details on the product line let me know and I can find a local team to speak with you. | *Simon Dodsley DD MBCS CITP* | Director, Open Source Integrations, WW Field Tech HQ | Pure Storage, Inc. | +1 (510) 516-2682 | simon@purestorage.com | www.purestorage.com PURE + OPENSTACK: Get more details here <https://www.purestorage.com/content/dam/pdf/en/solution-briefs/sb-enhancing-openstack-deployments-with-pure.pdf> On Mon, Feb 17, 2025 at 11:02 AM Rajat Dhasmana <rdhasman@redhat.com> wrote:
Hi Holger,
Looking at the cinder support matrix[1], only Pure storage has declared support for RoCE (I think they mean ROCEv2 only as RoCEv1 is rarely used) and os-brick does support RoCEv2. It should technically "work" with Infiniband hardware, as the main protocol is RDMA either over infiniband or ethernet cards but I have no experience trying it so better to ask the vendor i.e. Pure about it. Probably @Simon Dodsley <simon@purestorage.com> can provide more information on it.
[1] https://docs.openstack.org/cinder/latest/reference/support-matrix.html
Thanks Rajat Dhasmana
On Mon, Feb 17, 2025 at 7:09 PM Sean Mooney <smooney@redhat.com> wrote:
On 17/02/2025 12:58, holger.bielefeld@uponu.com wrote:
Hi all,
does anyone know if NVMe over RDMA is supported to attach it with Cinder/Nova to a QemuKVM Hypervisor?
technically yes.
it depend on if you cinder driver supports nvme over rdma as well.
ceph obviously does not as it use RBD to export the block storage.
the mounting of the NVMEoF devicve is handled by os-brick so form a nova/qemu point of view its a host mounted block device. as such there is no difference between that and iscsi because qemu is not directly connecting to the storage backend.
In addition to our ceph, we would also like to connect a block storage and in order to be able to retrieve the full performance, this must be done with the above-mentioned connection.
Thanks in advance.
Holger
participants (4)
-
holger.bielefeld＠uponu.com
-
Rajat Dhasmana
-
Sean Mooney
-
Simon Dodsley