<div dir="auto">Manila interacts with ceph using a ceph client user (this user is called “manila”by default, but is customizable with config option [1]). It’s likely that user doesn’t have permission to use the data pool that’s assigned to the underlying cephfs file system; can you check these permissions? (called “caps”)</div><div dir="auto"><br></div><div dir="auto">In Train, these are the permissions the ceph client user needs: <div><a href="https://docs.openstack.org/manila/train/admin/cephfs_driver.html#authorizing-the-driver-to-communicate-with-ceph">https://docs.openstack.org/manila/train/admin/cephfs_driver.html#authorizing-the-driver-to-communicate-with-ceph</a></div><div dir="auto"><br></div><div dir="auto">Can you check that these permissions match in your deployment?</div><div dir="auto"><br></div><div dir="auto">If they do, check the file permissions for the ceph keyring associated with the user. </div><div dir="auto"><br></div><div dir="auto">[1] <div><a href="https://github.com/openstack/manila/blob/191e8c9634d606c16ca0c1882cc01f4eb310d710/manila/share/drivers/cephfs/driver.py#L62">https://github.com/openstack/manila/blob/191e8c9634d606c16ca0c1882cc01f4eb310d710/manila/share/drivers/cephfs/driver.py#L62</a></div></div><div dir="auto"><br></div></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 16, 2022 at 6:08 PM Boxiang Zhu <<a href="mailto:bxzhu_5355@163.com">bxzhu_5355@163.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div style="line-height:1.7;font-size:14px;font-family:Arial;color:rgb(0,0,0)">
<div style="font-family:微软雅黑,Verdana,"Microsoft Yahei",SimSun,sans-serif;font-size:14px;line-height:1.6">
<div style="font-family:微软雅黑,Verdana,"Microsoft Yahei",SimSun,sans-serif"></div><div style="line-height:1.7;font-size:14px;font-family:Arial;color:rgb(0,0,0)"><p style="margin:0px;font-family:Arial"><br></p><div style="margin:0px;font-family:Arial">eblock, Thanks for your reply. I have read the report [1] but it can not fix my issue.</div><div style="margin:0px;font-family:Arial"><br></div><div style="margin:0px;font-family:Arial">In our manila-share container, the manila-share service is running by <b style="font-family:Arial">manila</b></div><div style="margin:0px;font-family:Arial"><b style="font-family:Arial">user not root user</b>.</div><div style="margin:0px;font-family:Arial"><br></div><div style="margin:0px;font-family:Arial">When I run as <b style="font-family:Arial">root user</b> into manila-share container with command like</div><div style="margin:0px;font-family:Arial">"docker exec -it -uroot manila_share bash". And then run "python" into</div><div style="margin:0px;font-family:Arial">interactive model. Then run code as followed:</div><div style="margin:0px;font-family:Arial"><div style="margin:0px;font-family:Arial"><div style="font-family:Consolas,"Courier New",monospace;line-height:19px;white-space:pre-wrap;background-color:rgb(30,30,30);color:rgb(212,212,212)"><div style="font-family:Consolas,"Courier New",monospace"><span style="font-family:Consolas,"Courier New",monospace;color:rgb(197,134,192)">import</span> <span style="font-family:Consolas,"Courier New",monospace;color:rgb(78,201,176)">ceph_volume_client</span></div><div style="font-family:Consolas,"Courier New",monospace"><span style="font-family:Consolas,"Courier New",monospace;color:rgb(156,220,254)">share</span>={<span style="font-family:Consolas,"Courier New",monospace;color:rgb(206,145,120)">"share_group_id"</span>:<span style="font-family:Consolas,"Courier New",monospace;color:rgb(86,156,214)">None</span>,<span style="font-family:Consolas,"Courier New",monospace;color:rgb(206,145,120)">"id"</span>:<span style="font-family:Consolas,"Courier New",monospace;color:rgb(206,145,120)">"449a52a4-f19c-4d1b-b437-7dc2443e040c"</span>}</div><div style="font-family:Consolas,"Courier New",monospace"><span style="font-family:Consolas,"Courier New",monospace;color:rgb(156,220,254)">c</span>=<span style="font-family:Consolas,"Courier New",monospace;color:rgb(78,201,176)">ceph_volume_client</span>.<span style="font-family:Consolas,"Courier New",monospace;color:rgb(78,201,176)">CephFSVolumeClient</span>(<span style="font-family:Consolas,"Courier New",monospace;color:rgb(206,145,120)">"manila"</span>, <span style="font-family:Consolas,"Courier New",monospace;color:rgb(206,145,120)">"/etc/ceph/ceph.conf"</span>, <span style="font-family:Consolas,"Courier New",monospace;color:rgb(206,145,120)">"ceph"</span>, <span style="font-family:Consolas,"Courier New",monospace;color:rgb(156,220,254)">volume_prefix</span>=<span style="font-family:Consolas,"Courier New",monospace;color:rgb(206,145,120)">"/volumes"</span>)</div><div style="font-family:Consolas,"Courier New",monospace"><span style="font-family:Consolas,"Courier New",monospace;color:rgb(156,220,254)">c</span>.<span style="font-family:Consolas,"Courier New",monospace;color:rgb(220,220,170)">connect</span>(<span style="font-family:Consolas,"Courier New",monospace;color:rgb(156,220,254)">premount_evict</span>=<span style="font-family:Consolas,"Courier New",monospace;color:rgb(206,145,120)">"manila"</span>)</div><div style="font-family:Consolas,"Courier New",monospace"><span style="font-family:Consolas,"Courier New",monospace;color:rgb(156,220,254)">v</span>=<span style="font-family:Consolas,"Courier New",monospace;color:rgb(156,220,254)">c</span>.<span style="font-family:Consolas,"Courier New",monospace;color:rgb(220,220,170)">create_volume</span>(<span style="font-family:Consolas,"Courier New",monospace;color:rgb(78,201,176)">ceph_volume_client</span>.<span style="font-family:Consolas,"Courier New",monospace;color:rgb(78,201,176)">VolumePath</span>(<span style="font-family:Consolas,"Courier New",monospace;color:rgb(156,220,254)">share</span>[<span style="font-family:Consolas,"Courier New",monospace;color:rgb(206,145,120)">'share_group_id'</span>], <span style="font-family:Consolas,"Courier New",monospace;color:rgb(156,220,254)">share</span>[<span style="font-family:Consolas,"Courier New",monospace;color:rgb(206,145,120)">'id'</span>]), <span style="font-family:Consolas,"Courier New",monospace;color:rgb(156,220,254)">size</span>=<span style="font-family:Consolas,"Courier New",monospace;color:rgb(181,206,168)">1073741824</span>, <span style="font-family:Consolas,"Courier New",monospace;color:rgb(156,220,254)">data_isolated</span>=<span style="font-family:Consolas,"Courier New",monospace;color:rgb(86,156,214)">False</span>, <span style="font-family:Consolas,"Courier New",monospace;color:rgb(156,220,254)">mode</span>=<span style="font-family:Consolas,"Courier New",monospace;color:rgb(78,201,176)">int</span>(<span style="font-family:Consolas,"Courier New",monospace;color:rgb(206,145,120)">'755'</span>,<span style="font-family:Consolas,"Courier New",monospace;color:rgb(181,206,168)">8</span>))</div></div></div></div><div style="margin:0px;font-family:Arial">I can <b style="font-family:Arial">succeed </b>to create the volume.</div><div style="margin:0px;font-family:Arial"><br></div><div style="margin:0px;font-family:Arial">But when I run as manila user into manila-share container with command</div><div style="margin:0px;font-family:Arial">like "docker exec -it manila_share bash". And then run the same python</div><div style="margin:0px;font-family:Arial">codes, it will raise the same error which appears in the manila-share.log file.</div><div style="margin:0px;font-family:Arial"><br></div><div style="margin:0px;font-family:Arial">[1]: <span style="font-family:arial;white-space:pre-wrap"><a href="https://bugs.launchpad.net/charm-manila-ganesha/+bug/1901570" target="_blank" style="font-family:arial">https://bugs.launchpad.net/charm-manila-ganesha/+bug/1901570</a></span></div><div style="zoom:1;font-family:Arial"></div><div id="m_-5646089457893048570divNeteaseMailCard" style="font-family:Arial"></div><p style="margin:0px;font-family:Arial"><br></p><p style="margin:0px;font-family:Arial"><span style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif;font-variant-numeric:normal;font-variant-east-asian:normal;line-height:22.4px;color:rgb(49,53,59)">Regards</span></p></div></div></div><div style="line-height:1.7;font-size:14px;font-family:Arial;color:rgb(0,0,0)"><div style="font-family:微软雅黑,Verdana,"Microsoft Yahei",SimSun,sans-serif;font-size:14px;line-height:1.6"><div style="line-height:1.7;font-size:14px;font-family:Arial;color:rgb(0,0,0)"><p style="margin:0px;font-family:Arial">Boxiang</p><pre style="font-family:monospace"><br>At 2022-02-16 21:41:15, "Eugen Block" <<a href="mailto:eblock@nde.ag" target="_blank" style="font-family:monospace">eblock@nde.ag</a>> wrote:
>This sounds a lot like this bug:
><a href="https://bugs.launchpad.net/charm-manila-ganesha/+bug/1901570" target="_blank" style="font-family:monospace">https://bugs.launchpad.net/charm-manila-ganesha/+bug/1901570</a>
>
>What are your ceph auth caps for the required pool(s)? I'm not
>familiar with Manila or Kolla, but maybe as a workaround you can
>change the directory permissions as described in the report.
>
>
>Zitat von Boxiang Zhu <<a href="mailto:bxzhu_5355@163.com" target="_blank" style="font-family:monospace">bxzhu_5355@163.com</a>>:
>
>> Hi all,
>>
>>
>> I have met an issue when I use manila with cephfs nfs backend.
>>
>>
>> Environment Information:
>> - OS: ubuntu 18.04
>> - OpenStack Version: Train
>> - Kolla-ansible Version: 9.3.2
>>
>>
>> I used kolla-ansible to deploy my OpenStack environment AIO.
>>
>>
>> Here is the info of my globals.yml as followed:
>> ---
>> kolla_base_distro: "ubuntu"
>> kolla_install_type: "source"
>> openstack_release: "train"
>> kolla_internal_vip_address: "172.16.150.67"
>> network_interface: "ens3"
>> neutron_external_interface: "ens4"
>> enable_haproxy: "no"
>> enable_ceph: "yes"
>> enable_ceph_mds: "yes"
>> enable_ceph_rgw: "yes"
>> enable_ceph_nfs: "yes"
>> enable_cinder: "yes"
>> enable_manila: "yes"
>> enable_manila_backend_cephfs_nfs: "yes"
>> enable_ceph_rgw_keystone: "yes"
>> ceph_osd_store_type: "filestore"
>>
>>
>> I try to use cephfs nfs backend for manila.
>> All containers are running and the services of manila is good.
>> +----+------------------+-------------------+------+---------+-------+----------------------------+
>> | Id | Binary | Host | Zone | Status | State | Updated_at |
>> +----+------------------+-------------------+------+---------+-------+----------------------------+
>> | 1 | manila-data | manila | nova | enabled | up |
>> 2022-02-16T03:25:34.000000 |
>> | 2 | manila-scheduler | manila | nova | enabled | up |
>> 2022-02-16T03:25:35.000000 |
>> | 3 | manila-share | manila@cephfsnfs1 | nova | enabled | up |
>> 2022-02-16T03:25:40.000000 |
>> +----+------------------+-------------------+------+---------+-------+----------------------------+
>>
>>
>> Here is my share type:
>> +--------------------------------------+--------------------+------------+------------+-------------------------------------+----------------------+-------------+
>> | ID | Name |
>> visibility | is_default | required_extra_specs |
>> optional_extra_specs | Description |
>> +--------------------------------------+--------------------+------------+------------+-------------------------------------+----------------------+-------------+
>> | 265a6637-0322-4c4a-9185-f30f01b96d12 | default_share_type | public
>> | - | driver_handles_share_servers : False |
>> | None |
>> +--------------------------------------+--------------------+------------+------------+-------------------------------------+----------------------+-------------+
>>
>>
>> When I create a share, error information appears in manila-share.log file:
>>
>>
>> 2022-02-16 09:30:04.615 20 INFO manila.share.manager
>> [req-b57991af-5355-4202-8543-dc7ff914d919 - - - - -] Updating share
>> status
>> 2022-02-16 09:30:04.616 20 DEBUG manila.share.driver
>> [req-b57991af-5355-4202-8543-dc7ff914d919 - - - - -] Updating share
>> stats. _update_share_stats
>> /var/lib/kolla/venv/lib/python3.6/site-packages/manila/share/driver.py:1232
>> 2022-02-16 09:30:54.675 20 DEBUG manila.share.drivers.cephfs.driver
>> [req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b
>> eaf0e30a92694889aa46ac5a1d4b7a47 37025e7c9ae447c8975e9ef3a4e5d0ff -
>> - -] create_share CEPHFSNFS1
>> name=449a52a4-f19c-4d1b-b437-7dc2443e040c size=1 share_group_id=None
>> create_share
>> /var/lib/kolla/venv/lib/python3.6/site-packages/manila/share/drivers/cephfs/driver.py:262
>> 2022-02-16 09:30:54.682 20 INFO ceph_volume_client
>> [req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b
>> eaf0e30a92694889aa46ac5a1d4b7a47 37025e7c9ae447c8975e9ef3a4e5d0ff -
>> - -] create_volume:
>> /volumes/_nogroup/449a52a4-f19c-4d1b-b437-7dc2443e040c2022-02-16
>> 09:30:54.927 20 ERROR manila.share.manager
>> [req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b
>> eaf0e30a92694889aa46ac5a1d4b7a47 37025e7c9ae447c8975e9ef3a4e5d0ff -
>> - -] Share instance 449a52a4-f19c-4d1b-b437-7dc2443e040c failed on
>> creation.: cephfs.OSError: error in mkdir
>> volumes/_nogroup/449a52a4-f19c-4d1b-b437-7dc2443e040c: Permission
>> denied [Errno 13]
>> 2022-02-16 09:30:54.929 20 WARNING manila.share.manager
>> [req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b
>> eaf0e30a92694889aa46ac5a1d4b7a47 37025e7c9ae447c8975e9ef3a4e5d0ff -
>> - -] Share instance information in exception can not be written to
>> db because it contains {} and it is not a dictionary.:
>> cephfs.OSError: error in mkdir
>> volumes/_nogroup/449a52a4-f19c-4d1b-b437-7dc2443e040c: Permission
>> denied [Errno 13]
>> 2022-02-16 09:30:54.955 20 INFO manila.message.api
>> [req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b
>> eaf0e30a92694889aa46ac5a1d4b7a47 37025e7c9ae447c8975e9ef3a4e5d0ff -
>> - -] Creating message record for request_id =
>> req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
>> [req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b
>> eaf0e30a92694889aa46ac5a1d4b7a47 37025e7c9ae447c8975e9ef3a4e5d0ff -
>> - -] Exception during message handling: cephfs.OSError: error in
>> mkdir volumes/_nogroup/449a52a4-f19c-4d1b-b437-7dc2443e040c:
>> Permission denied [Errno 13]
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server Traceback
>> (most recent call last):
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File
>> "/usr/lib/python3/dist-packages/ceph_volume_client.py", line 656, in
>> _mkdir_p
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
>> self.fs.stat(subpath)
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File
>> "cephfs.pyx", line 1257, in cephfs.LibCephFS.stat
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
>> cephfs.ObjectNotFound: error in stat:
>> volumes/_nogroup/449a52a4-f19c-4d1b-b437-7dc2443e040c: No such file
>> or directory [Errno 2]
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server During
>> handling of the above exception, another exception occurred:
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server Traceback
>> (most recent call last):
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in
>> _process_incoming
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server res =
>> self.dispatcher.dispatch(message)
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in
>> dispatch
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
>> return self._do_dispatch(endpoint, method, ctxt, args)
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in
>> _do_dispatch
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
>> result = func(ctxt, **new_args)
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/manila/share/manager.py",
>> line 187, in wrapped
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
>> return f(self, *args, **kwargs)
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/manila/utils.py",
>> line 568, in wrapper
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
>> return func(self, *args, **kwargs)
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/manila/share/manager.py",
>> line 1790, in create_share_instance
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server exception=e)
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 220, in __exit__
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
>> self.force_reraise()
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 196, in force_reraise
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
>> six.reraise(self.type_, self.value, self.tb)
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/six.py", line 693,
>> in reraise
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server raise value
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/manila/share/manager.py",
>> line 1753, in create_share_instance
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
>> context, share_instance, share_server=share_server)
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/manila/share/drivers/cephfs/driver.py", line 272, in
>> create_share
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
>> mode=self._cephfs_volume_mode)
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File
>> "/usr/lib/python3/dist-packages/ceph_volume_client.py", line 677, in
>> create_volume
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
>> self._mkdir_p(path, mode)
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File
>> "/usr/lib/python3/dist-packages/ceph_volume_client.py", line 658, in
>> _mkdir_p
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
>> self.fs.mkdir(subpath, mode)
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File
>> "cephfs.pyx", line 887, in cephfs.LibCephFS.mkdir
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
>> cephfs.OSError: error in mkdir
>> volumes/_nogroup/449a52a4-f19c-4d1b-b437-7dc2443e040c: Permission
>> denied [Errno 13]
>> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
>
>
>
</pre><br></div>
</div>
</div><br><br><span title="neteasefooter"><p> </p></span></blockquote></div></div>