Hi, I redid the configuration as mentioned and I used my nfs cluster ID. The share was created, the access right does not show the error status and df -h shows the nfs mount, but still the filesystem is read-only.... [root@controllera ~]# ceph nfs cluster create mynfs "3 controllera controllerb controllerc" --ingress --virtual_ip 20.1.0.201 --ingress-mode haproxy-protocol [root@controllera ~]# ceph orch ls | grep nfs ingress.nfs.mynfs 20.1.0.201:2049,9049 6/6 5m ago 4d controllera;controllerb;controllerc;count:3 *nfs.mynfs ?:12049 3/3 5m ago 4d controllera;controllerb;controllerc;count:3* cat *manila.conf* ..... [cephfsnfs1] driver_handles_share_servers = False share_backend_name = CEPHFSNFS1 share_driver = manila.share.drivers.cephfs.driver.CephFSDriver cephfs_protocol_helper_type = NFS cephfs_conf_path = /etc/ceph/ceph.conf cephfs_auth_id = manila cephfs_cluster_name = ceph cephfs_filesystem_name = cephfs *cephfs_nfs_cluster_id = mynfs* [root@rcdndeployer2 ~]# cat /etc/kolla2023dot2/config/manila/ceph.client.manila.keyring [client.manila] key = AQCra99lC3DTAhAA5Wk+trH9dc/1TIIv3SId1w== caps mgr = "allow rw" caps mon = "allow r" (2023.2) [deployer@rcdndeployer2 ~]$ openstack share list +--------------------------------------+------------------+------+-------------+-----------+-----------+------------------+----------------------------------+-------------------+ | ID | Name | Size | Share Proto | Status | Is Public | Share Type Name | Host | Availability Zone | +--------------------------------------+------------------+------+-------------+-----------+-----------+------------------+----------------------------------+-------------------+ | 9b25edbb-4aa3-401e-89d1-6f1f130e62c3 | cephnativeshare1 | 3 | CEPHFS | available | False | cephfsnativetype | controllerb@cephfsnative1#cephfs | nova | |* 9aa19f2a-7b7e-4274-b22b-5a2a76e0f4a2 | cephnfsshare4 | 4 | NFS | available | False | cephfsnfstype | controllera@cephfsnfs1#cephfs | nova |* +--------------------------------------+------------------+------+-------------+-----------+-----------+------------------+----------------------------------+------- (2023.2) [deployer@rcdndeployer2 ~]$ openstack share export location list cephnfsshare4 +--------------------------------------+--------------------------------------------------------------------------------------------------------+-----------+ | ID | Path | Preferred | +--------------------------------------+--------------------------------------------------------------------------------------------------------+-----------+ | 8c614f22-c221-45f9-960d-b85b8b3de94c | 20.1.0.201:/volumes/_nogroup/f7a9b751-f013-4a8b-8ee9-24da834c6da7/fba335fe-7122-4b2c-a955-c3884ce79e42 | False | <-------- my storage net 20.1.0.0/16 | 31ed7a75-0537-46d5-9b0b-1d2a1336b10b | 20.3.0.23:/volumes/_nogroup/f7a9b751-f013-4a8b-8ee9-24da834c6da7/fba335fe-7122-4b2c-a955-c3884ce79e42 | False | <-------- my api net (I don't know why the share was exported on this network ????) +--------------------------------------+--------------------------------------------------------------------------------------------------------+-----------+ (2023.2) [deployer@rcdndeployer2 ~]$ openstack share access list cephnfsshare4 +--------------------------------------+-------------+------------+--------------+--------+------------+----------------------------+----------------------------+ | ID | Access Type | Access To | Access Level | State | Access Key | Created At | Updated At | +--------------------------------------+-------------+------------+--------------+--------+------------+----------------------------+----------------------------+ | ccdbfba5-691a-4591-93ee-cef30facc5d5 | ip | *20.1.11.29 | rw* | active | None | 2024-03-07T13:19:29.199046 | 2024-03-07T13:19:29.487724 | +--------------------------------------+-------------+------------+--------------+--------+------------+----------------------------+----------------------------+ From the client machine [root@alicer9manila ~]# mount -t nfs4 20.1.0.201:/volumes/_nogroup/f7a9b751-f013-4a8b-8ee9-24da834c6da7/fba335fe-7122-4b2c-a955-c3884ce79e42 /mnt/nfscephfs/ [root@alicer9manila ~]# df -h | grep nfs 20.1.0.201:/volumes/_nogroup/f7a9b751-f013-4a8b-8ee9-24da834c6da7/fba335fe-7122-4b2c-a955-c3884ce79e42 4.0G 0 4.0G 0% /mnt/nfscephfs [root@alicer9manila ~]# touch /mnt/nfscephfs/file *touch: cannot touch '/mnt/nfscephfs/file': Read-only file system* Any ideas? Regards. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> Le jeu. 7 mars 2024 à 02:02, Goutham Pacha Ravi <gouthampravi@gmail.com> a écrit :
Hello again,
Documentation for this is being added here: https://review.opendev.org/c/openstack/manila/+/911632 Would welcome your feedback!
Thanks!
On Wed, Mar 6, 2024 at 3:15 PM Goutham Pacha Ravi <gouthampravi@gmail.com> wrote:
On Wed, Mar 6, 2024 at 2:32 PM wodel youchi <wodel.youchi@gmail.com>
Hi,
Thanks for your help.
I've created an NFS service on my ceph cluster with haproxy and
keepalived, the VIP is 20.1.0.201
I've created this keyring for the Manila client : [client.manila] key = AQCra99lC3DT*************88Iv3SId1w== caps mgr = "allow rw" caps mon = "allow r" caps osd = "allow rw pool=.nfs" <------------------------ Should I keep this line
No, it isn't necessary. the CephFS/NFS driver (using this user/keyring) doesn't directly manipulate anything in the ".nfs" pool. So you can remove this osd capability
Should I keep the last line about the .nfs pool?
I added the NFS driver to manila service using the documentation cat /etc/kolla/manila-api/manila.conf ... [cephfsnfs1] driver_handles_share_servers = False share_backend_name = CEPHFSNFS1 share_driver = manila.share.drivers.cephfs.driver.CephFSDriver cephfs_protocol_helper_type = NFS cephfs_conf_path = /etc/ceph/ceph.conf cephfs_auth_id = manila cephfs_cluster_name = ceph cephfs_filesystem_name = cephfs cephfs_ganesha_server_is_remote = False cephfs_ganesha_server_ip = 20.1.0.201 ganesha_rados_store_enable = True ganesha_rados_store_pool_name = .nfs
<----------------------------------- Is this line correct, should I keep it?
No, remove "cephfs_ganesha_server_is_remote", "cephfs_ganesha_server_ip", "ganesha_rados_store_enable", "ganesha_rados_store_pool_name" options and replace them with this option
"cephfs_nfs_cluster_id=<name/id of the NFS service created via cephadm>"
for example, if you created a cluster like this: "ceph nfs cluster create mycephfsnfscluster ...", the ID would be "mycephfsnfscluster"
I've created an NFS share and it seems it has been created Openstack (2023.2) [deployer@rcdndeployer2 ~]$ openstack share export location
+--------------------------------------+--------------------------------------------------------------------------------------------------------+-----------+
| ID | Path | Preferred |
+--------------------------------------+--------------------------------------------------------------------------------------------------------+-----------+
| 4c5f5c4b-8308-40e5-9ee9-afed9f99a257 | 20.1.0.201:/volumes/_nogroup/bf98708e-69a4-4544-902b-fe8bd18d0d99/909330f4-9287-4af7-95eb-44f66e6e14dc | False |
+--------------------------------------+--------------------------------------------------------------------------------------------------------+-----------+
Ceph [root@controllera ~]# ceph fs subvolume ls cephfs [ { "name": "2bc1651d-a52b-44cc-b50d-7eab9fab35ed" }, { "name": "bf98708e-69a4-4544-902b-fe8bd18d0d99" } ] [root@controllera ~]# ceph fs subvolume getpath cephfs
bf98708e-69a4-4544-902b-fe8bd18d0d99
/volumes/_nogroup/bf98708e-69a4-4544-902b-fe8bd18d0d99/909330f4-9287-4af7-95eb-44f66e6e14dc
I gave it an access right, but I got an error Openstack (2023.2) [deployer@rcdndeployer2 ~]$ openstack share access list
cephnfsshare4
+--------------------------------------+-------------+------------+--------------+-------+------------+----------------------------+----------------------------+
| ID | Access Type | Access To | Access Level | State | Access Key | Created At | Updated At |
+--------------------------------------+-------------+------------+--------------+-------+------------+----------------------------+----------------------------+
| 8472c761-1d8f-4730-91a6-864f2c9cf85d | ip | 20.1.11.29 | rw | error | None | 2024-03-05T17:23:57.186732 | 2024-03-05T17:23:59.261406 |
+--------------------------------------+-------------+------------+--------------+-------+------------+----------------------------+----------------------------+
After you've made the configuration changes above, restart your manila-share manager service, delete and re-apply this rule and try your mount commands again
When I tried to mount the share I got several problems
From the client VM (IP 20.1.11.26 /16)
- Mounting using the volume path didnt't work I got : root@alicer9manila ~]# mount -v -t nfs4 20.1.0.201:/volumes/_nogroup/bf98708e-69a4-4544-902b-fe8bd18d0d99/909330f4-9287-4af7-95eb-44f66e6e14dc
/mnt/nfscephfs/
mount.nfs4: timeout set for Wed Mar 6 22:21:12 2024 mount.nfs4: trying text-based options 'vers=4.2,addr=20.1.0.201,clientaddr=20.1.11.29' mount.nfs4: mount(2): No such file or directory mount.nfs4: mounting 20.1.0.201:/volumes/_nogroup/bf98708e-69a4-4544-902b-fe8bd18d0d99/909330f4-9287-4af7-95eb-44f66e6e14dc failed, reason given by server: No such file or directory
So I used the root directory [root@alicer9manila ~]# mount -v -t nfs4 20.1.0.201:/ /mnt/nfscephfs/ mount.nfs4: timeout set for Wed Mar 6 22:22:13 2024 mount.nfs4: trying text-based options 'vers=4.2,addr=20.1.0.201,clientaddr=20.1.11.29'
- df -h : don't show the share [root@alicer9manila ~]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 4.0M 0 4.0M 0% /dev tmpfs 882M 0 882M 0% /dev/shm tmpfs 353M 5.0M 348M 2% /run /dev/vda5 19G 1.5G 18G 8% / /dev/vda2 994M 271M 724M 28% /boot /dev/vda1 100M 7.0M 93M 7% /boot/efi tmpfs 177M 0 177M 0% /run/user/1000
- mount shows that there is a mounted share [root@alicer9manila ~]# mount | grep nfs rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) 20.1.0.201:/ on /mnt/nfscephfs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=20.1.11.29,local_lock=none,addr=20.1.0.201)
- When trying to create a file on the mount-point I got : FS Read Only [root@alicer9manila ~]# cd /mnt/nfscephfs/ [root@alicer9manila nfscephfs]# touch file touch: cannot touch 'file': Read-only file system
How can I debug this?
Regards.
Le mer. 6 mars 2024 à 16:36, Carlos Silva <ces.eduardo98@gmail.com> a écrit :
Em dom., 3 de mar. de 2024 às 08:49, wodel youchi <
wodel.youchi@gmail.com> escreveu:
Hi,
I am having trouble understanding the documentation.
Kolla-ansible documentation about manila and cephfs-nfs say this : Prerequisites
3.0 or later versions of NFS-Ganesha. NFS client installed in the guest. Network connectivity between your Ceph cluster’s public network and
NFS-Ganesha server. <-------------------------- NFS Ganesha is an external service.
Yes, it is an external service. We introduced the ability to use the cephadm deployed ganesha a couple of releases ago, and started recommending it, but some docs still have to be updated, which is the case for this doc. Some of the steps are not necessary anymore and the amount of things you'd need to do in the setup is smaller.
Network connectivity between your NFS-Ganesha server and the manila
guest.
Ceph documentation says, that ceph can create an NFS-Ganesha
cluster, which simplifies the deployment, but...there are some things that are clear :
- When ceph create the NFS cluster, it create a new pool, in my deployment it is called ".nfs" - Manila using cephfs-nfs needs a pool to save the export configuration, which is called for reference : <ganesha_rados_store_pool_name> Is it the same pool???
You don't need to set this option using a ganesha deployed by cephadm, or even create the pool as suggested. That will be already done.
- The Ganesha configuration says : For a fresh setup, make sure to
create the Ganesha export index object as an empty object before starting
echo | sudo rados -p ${GANESHA_RADOS_STORE_POOL_NAME} put ganesha-export-index - - If its the .nfs pool, did ceph put the ganesha-export-index in it already or should I do it?
No need for this ganesha export index part either, it will already be
wrote: list cephnfsshare4 the Ganesha server. figured out in this case.
Thanks.
Le ven. 1 mars 2024 à 22:04, Carlos Silva <ces.eduardo98@gmail.com>
a écrit :
Hello,
Em sex., 1 de mar. de 2024 às 14:09, wodel youchi <
wodel.youchi@gmail.com> escreveu:
> > Hi, > > I am trying to configure Manila with cephsfs NFS as backend. > My first problem is with installing and configuring nfs ganesha : > - Do I have to deploy nfs ganesha from ceph?
Yeah, I think the most viable option at this point would be to deploy ganesha using cephadm. The Manila Ceph NFS driver supports it and you'll get good benefits using this approach. You can find some guidance on how to deploy NFS Ganesha with cephadm this doc [0]. > > - What about the /etc/ganesha/ganesha.conf config file?
This Manila documentation [1] can be useful for understanding the config files. > > > Could someone give me some steps to follow... > > Regards.
[0] https://docs.ceph.com/en/latest/cephadm/services/nfs/ [1] https://docs.openstack.org/manila/latest/contributor/ganesha.html#nfs-ganesh...