Open Stack

Hi Jeff,

Currently, there is no open source backend in manila that provides
scalable and highly-available NFS servers for dynamic cloud workloads.
Manila's CephFS driver could integrate with your on-going work
on active-active NFS over CephFS (with Kubernetes managing the
lifecycle of containerized user-space NFS-Ganesha servers) [1]
to fill this gap.

During the manila project team gathering, we discussed this plan
under the topic of high availability of share servers [2]. One of the
questions was about write integrity issue when a NFS-Ganesha server
container goes down and another container comes up to replace it.
Would there be any such write integrity issues when NFS clients do
asynchronous writes to files with write caching in the NFS client
and the NFS server (NFS-Ganesha server side caching or the libcephfs client
caching), and the NFS-Ganesha server goes down? I guess this a general
NFS protocol question or maybe things get complicated with NFS-Ganesha over CephFS?

I looked up the NFSv4 protocol documentation, the implementation of COMMIT
operation [3]. So if a NFS client issues a aysnc write followed by a
COMMIT operation that succeeds, then it's expected that the NFS
server has flushed cached data and metadata onto stable storage,
here CephFS. And if the NFS server crashes losing cached data and metadata,
then the write verifier cookie returned by the WRITE or COMMIT operation
indicates to the client that the server crashed. Now it's up to the
NFS client to re-transmit the uncached data and metadata.

Thanks,
Ramana

[1] https://jtlayton.wordpress.com/2017/11/07/active-active-nfs-over-cephfs/

[2] line 186 in https://etherpad.openstack.org/p/manila-rocky-ptg
    the actual spec https://review.openstack.org/#/c/504987/

[3] https://tools.ietf.org/html/rfc7530#section-16.3.5