Hi Garcetto,
This is a nice use case; but no.. it's not supported.
IIUC you're using Manila's generic driver, and cinder is backed by rbd. Each generic driver share server (NFS server) thinks it is working in isolation, and will not use pre-existing cinder volumes to create shares. It may be something you can work around in the code though. The next inconvenience is that the NFS servers are different, and they're not "clustered"; which means that you don't get the ability to coordinate reads/writes amongst the three NFS servers. So you could run into data corruption issues. The third problem is with multi-attach semantics: the same cinder volume needs to be attached to (and exported via) three different NFS servers. I think EFS has these smarts figured out to enable coordinated simultaneous access to the same underlying filesystem from different front-end NFS servers.
The closest I can think of is clustered NFS-Ganesha and the CephFS driver in Manila. Ceph (since Quincy) has a capability now to deploy active-active NFS clusters with cephadm (a service called "ceph-nfs"). You can layer an ingress in front of this cluster if you like; or wire up each component NFS server on an AZ specific subnet. This way, a single share has multiple export paths, one in each subnet/AZ - and since the NFS service is clustered, you're guaranteed coherence and durability, like EFS's "Regional file systems" [2]. Manila supports CephFS with the clustered NFS service since the "zed" release of OpenStack. [3]
[1] https://docs.ceph.com/en/reef/cephadm/services/nfs/ [2] https://docs.aws.amazon.com/efs/latest/ug/how-it-works.html#how-it-works-ec2
[3] https://blueprints.launchpad.net/manila/+spec/use-cephadm-nfs-ganesha
On Thu, Feb 8, 2024 at 7:35 AM garcetto garcetto@gmail.com wrote:
good afternoon, i am using zed, on streched ceph as backend on multiple az. i would like to emulate aws efs multiple nfs mount points for each user tenant (one on each different az) using manila, is it possible? ideally i would like to expose to each users tenant a manila vm as nfs exporter to custormer' instances, and as backend (of manila) use a cinder volume on ceph (rbd? cephfs? nfs?) my problem is how can i create say 3 different manila service vms (one in each az) with different frontend ips all pointing to same rbd/cephfs volume? am i doing wrong?
thank you.