hello, I'm trying to install manila in my openstack cloud (deployed on k8s cluster) but having problem. 1 -> i created the auth using ceph auth get-or-create client.manila -o manila.keyring \ mgr 'allow rw' \ mon 'allow r' source - https://docs.openstack.org/manila/latest/configuration/shared-file-systems/d... 2 -> in manila-share pod i see the error 2024-10-08 07:37:25.090 7 INFO manila.share.drivers.cephfs.driver [None req-fffe12f7-c7aa-42dc-aea1-bed6f918b278 - - - - - -] [CEPHFSNATIVE1] Ceph client found, connecting... 2024-10-08 07:37:25.104 7 ERROR manila.share.manager [None req-fffe12f7-c7aa-42dc-aea1-bed6f918b278 - - - - - -] Error encountered during initialization of driver CephFSDriver@lab-11053-26006-ceph-1@cephfsnative1: manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. 2024-10-08 07:37:25.104 7 ERROR manila.share.manager Traceback (most recent call last): 2024-10-08 07:37:25.104 7 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 421, in rados_client 2024-10-08 07:37:25.104 7 ERROR manila.share.manager self._rados_client.connect() 2024-10-08 07:37:25.104 7 ERROR manila.share.manager File "rados.pyx", line 690, in rados.Rados.connect 2024-10-08 07:37:25.104 7 ERROR manila.share.manager rados.IOError: [errno 5] RADOS I/O error (error connecting to the cluster) 2024-10-08 07:37:25.104 7 ERROR manila.share.manager 2024-10-08 07:37:25.104 7 ERROR manila.share.manager During handling of the above exception, another exception occurred: 2024-10-08 07:37:25.104 7 ERROR manila.share.manager 2024-10-08 07:37:25.104 7 ERROR manila.share.manager Traceback (most recent call last): 2024-10-08 07:37:25.104 7 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/manager.py", line 371, in _driver_setup 2024-10-08 07:37:25.104 7 ERROR manila.share.manager self.driver.do_setup(ctxt) 2024-10-08 07:37:25.104 7 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 274, in do_setup 2024-10-08 07:37:25.104 7 ERROR manila.share.manager rados_client=self.rados_client, 2024-10-08 07:37:25.104 7 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 424, in rados_client 2024-10-08 07:37:25.104 7 ERROR manila.share.manager raise exception.ShareBackendException( 2024-10-08 07:37:25.104 7 ERROR manila.share.manager manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. 2024-10-08 07:37:25.104 7 ERROR manila.share.manager 2024-10-08 07:37:25.105 7 DEBUG manila.utils [None req-fffe12f7-c7aa-42dc-aea1-bed6f918b278 - - - - - -] Finished call to 'manila.share.manager.ShareManager.init_host.<locals>._driver_setup' after 62.106(s), this was the 6th time calling it. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/after.py:44 2024-10-08 07:37:25.105 7 DEBUG manila.utils [None req-fffe12f7-c7aa-42dc-aea1-bed6f918b278 - - - - - -] Retrying manila.share.manager.ShareManager.init_host.<locals>._driver_setup in 64.0 seconds as it raised ShareBackendException: [%(be)s] Ceph client failed to connect.. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/before_sleep.py:65 3 -> when i try to use the maila keyring i get the error *[rook@cmp-2 ceph]$ ceph --key AQBip/9mCMf4N/jFNKFlwl/hA4j64esWuA== osd ls[errno 13] RADOS permission denied (error connecting to the cluster)*this is what i get after running that not sure what permissions any idea? -> i also tried with admin rights to the auth *[rook@cmp-2 ceph]$ ceph auth get client.manila[client.manila] key = AQBip/9mCMf4N/jFNKFlwl/hA4j64esWuA== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *"exported keyring for client.manila * 4-> my manila conf looks like conf: manila: DEFAULT: debug: true enabled_share_protocols: NFS,CIFS,CEPHFS enabled_share_backends: cephfsnative1 cephfsnative1: driver_handles_share_servers: False share_backend_name: CEPHFSNATIVE1 share_driver: manila.share.drivers.cephfs.driver.CephFSDriver cephfs_conf_path: /etc/ceph/ceph.conf cephfs_protocol_helper_type: CEPHFS cephfs_auth_id: manila cephfs_cluster_name: rook-ceph cephfs_filesystem_name: myfs any idea what could be wrong? Hope to hear from you soon Thank you!