Open Stack

Manila interacts with ceph using a ceph client user (this user is called
“manila”by default, but is customizable with config option [1]). It’s
likely that user doesn’t have permission to use the data pool that’s
assigned to the underlying cephfs file system; can you check these
permissions? (called “caps”)

In Train, these are the permissions the ceph client user needs:
https://docs.openstack.org/manila/train/admin/cephfs_driver.html#authorizing-the-driver-to-communicate-with-ceph

Can you check that these permissions match in your deployment?

If they do, check the file permissions for the ceph keyring associated with
the user.

[1]
https://github.com/openstack/manila/blob/191e8c9634d606c16ca0c1882cc01f4eb310d710/manila/share/drivers/cephfs/driver.py#L62


On Wed, Feb 16, 2022 at 6:08 PM Boxiang Zhu <bxzhu_5355 at 163.com> wrote:

>
> eblock, Thanks for your reply. I have read the report [1] but it can not
> fix my issue.
>
> In our manila-share container, the manila-share service is running by
> *manila*
> *user not root user*.
>
> When I run as *root user* into manila-share container with command like
> "docker exec -it -uroot manila_share bash". And then run "python" into
> interactive model. Then run code as followed:
> import ceph_volume_client
> share={"share_group_id":None,"id":"449a52a4-f19c-4d1b-b437-7dc2443e040c"}
> c=ceph_volume_client.CephFSVolumeClient("manila", "/etc/ceph/ceph.conf",
> "ceph", volume_prefix="/volumes")
> c.connect(premount_evict="manila")
> v=c.create_volume(ceph_volume_client.VolumePath(share['share_group_id'],
> share['id']), size=1073741824, data_isolated=False, mode=int('755',8))
> I can *succeed *to create the volume.
>
> But when I run as manila user into manila-share container with command
> like "docker exec -it manila_share bash". And then run the same python
> codes, it will raise the same error which appears in the manila-share.log
> file.
>
> [1]: https://bugs.launchpad.net/charm-manila-ganesha/+bug/1901570
>
>
> Regards
>
> Boxiang
>
>
> At 2022-02-16 21:41:15, "Eugen Block" <eblock at nde.ag> wrote:
> >This sounds a lot like this bug:
> >https://bugs.launchpad.net/charm-manila-ganesha/+bug/1901570
> >
> >What are your ceph auth caps for the required pool(s)? I'm not
> >familiar with Manila or Kolla, but maybe as a workaround you can
> >change the directory permissions as described in the report.
> >
> >
> >Zitat von Boxiang Zhu <bxzhu_5355 at 163.com>:
> >
> >> Hi all,
> >>
> >>
> >> I have met an issue when I use manila with cephfs nfs backend.
> >>
> >>
> >> Environment Information:
> >> - OS: ubuntu 18.04
> >> - OpenStack Version: Train
> >> - Kolla-ansible Version: 9.3.2
> >>
> >>
> >> I used kolla-ansible to deploy my OpenStack environment AIO.
> >>
> >>
> >> Here is the info of my globals.yml as followed:
> >> ---
> >> kolla_base_distro: "ubuntu"
> >> kolla_install_type: "source"
> >> openstack_release: "train"
> >> kolla_internal_vip_address: "172.16.150.67"
> >> network_interface: "ens3"
> >> neutron_external_interface: "ens4"
> >> enable_haproxy: "no"
> >> enable_ceph: "yes"
> >> enable_ceph_mds: "yes"
> >> enable_ceph_rgw: "yes"
> >> enable_ceph_nfs: "yes"
> >> enable_cinder: "yes"
> >> enable_manila: "yes"
> >> enable_manila_backend_cephfs_nfs: "yes"
> >> enable_ceph_rgw_keystone: "yes"
> >> ceph_osd_store_type: "filestore"
> >>
> >>
> >> I try to use cephfs nfs backend for manila.
> >> All containers are running and the services of manila is good.
> >> +----+------------------+-------------------+------+---------+-------+----------------------------+
> >> | Id | Binary | Host | Zone | Status | State | Updated_at |
> >> +----+------------------+-------------------+------+---------+-------+----------------------------+
> >> | 1 | manila-data | manila | nova | enabled | up |
> >> 2022-02-16T03:25:34.000000 |
> >> | 2 | manila-scheduler | manila | nova | enabled | up |
> >> 2022-02-16T03:25:35.000000 |
> >> | 3 | manila-share | manila at cephfsnfs1 | nova | enabled | up |
> >> 2022-02-16T03:25:40.000000 |
> >> +----+------------------+-------------------+------+---------+-------+----------------------------+
> >>
> >>
> >> Here is my share type:
> >> +--------------------------------------+--------------------+------------+------------+-------------------------------------+----------------------+-------------+
> >> | ID                                   | Name               |
> >> visibility | is_default | required_extra_specs                |
> >> optional_extra_specs | Description |
> >> +--------------------------------------+--------------------+------------+------------+-------------------------------------+----------------------+-------------+
> >> | 265a6637-0322-4c4a-9185-f30f01b96d12 | default_share_type | public
> >>     | -          | driver_handles_share_servers : False |
> >>           | None        |
> >> +--------------------------------------+--------------------+------------+------------+-------------------------------------+----------------------+-------------+
> >>
> >>
> >> When I create a share, error information appears in manila-share.log file:
> >>
> >>
> >> 2022-02-16 09:30:04.615 20 INFO manila.share.manager
> >> [req-b57991af-5355-4202-8543-dc7ff914d919 - - - - -] Updating share
> >> status
> >> 2022-02-16 09:30:04.616 20 DEBUG manila.share.driver
> >> [req-b57991af-5355-4202-8543-dc7ff914d919 - - - - -] Updating share
> >> stats. _update_share_stats
> >> /var/lib/kolla/venv/lib/python3.6/site-packages/manila/share/driver.py:1232
> >> 2022-02-16 09:30:54.675 20 DEBUG manila.share.drivers.cephfs.driver
> >> [req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b
> >> eaf0e30a92694889aa46ac5a1d4b7a47 37025e7c9ae447c8975e9ef3a4e5d0ff -
> >> - -] create_share CEPHFSNFS1
> >> name=449a52a4-f19c-4d1b-b437-7dc2443e040c size=1 share_group_id=None
> >> create_share
> >> /var/lib/kolla/venv/lib/python3.6/site-packages/manila/share/drivers/cephfs/driver.py:262
> >> 2022-02-16 09:30:54.682 20 INFO ceph_volume_client
> >> [req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b
> >> eaf0e30a92694889aa46ac5a1d4b7a47 37025e7c9ae447c8975e9ef3a4e5d0ff -
> >> - -] create_volume:
> >> /volumes/_nogroup/449a52a4-f19c-4d1b-b437-7dc2443e040c2022-02-16
> >> 09:30:54.927 20 ERROR manila.share.manager
> >> [req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b
> >> eaf0e30a92694889aa46ac5a1d4b7a47 37025e7c9ae447c8975e9ef3a4e5d0ff -
> >> - -] Share instance 449a52a4-f19c-4d1b-b437-7dc2443e040c failed on
> >> creation.: cephfs.OSError: error in mkdir
> >> volumes/_nogroup/449a52a4-f19c-4d1b-b437-7dc2443e040c: Permission
> >> denied [Errno 13]
> >> 2022-02-16 09:30:54.929 20 WARNING manila.share.manager
> >> [req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b
> >> eaf0e30a92694889aa46ac5a1d4b7a47 37025e7c9ae447c8975e9ef3a4e5d0ff -
> >> - -] Share instance information in exception can not be written to
> >> db because it contains {} and it is not a dictionary.:
> >> cephfs.OSError: error in mkdir
> >> volumes/_nogroup/449a52a4-f19c-4d1b-b437-7dc2443e040c: Permission
> >> denied [Errno 13]
> >> 2022-02-16 09:30:54.955 20 INFO manila.message.api
> >> [req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b
> >> eaf0e30a92694889aa46ac5a1d4b7a47 37025e7c9ae447c8975e9ef3a4e5d0ff -
> >> - -] Creating message record for request_id =
> >> req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
> >> [req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b
> >> eaf0e30a92694889aa46ac5a1d4b7a47 37025e7c9ae447c8975e9ef3a4e5d0ff -
> >> - -] Exception during message handling: cephfs.OSError: error in
> >> mkdir volumes/_nogroup/449a52a4-f19c-4d1b-b437-7dc2443e040c:
> >> Permission denied [Errno 13]
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server Traceback
> >> (most recent call last):
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server   File
> >> "/usr/lib/python3/dist-packages/ceph_volume_client.py", line 656, in
> >> _mkdir_p
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
> >> self.fs.stat(subpath)
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server   File
> >> "cephfs.pyx", line 1257, in cephfs.LibCephFS.stat
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
> >> cephfs.ObjectNotFound: error in stat:
> >> volumes/_nogroup/449a52a4-f19c-4d1b-b437-7dc2443e040c: No such file
> >> or directory [Errno 2]
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server During
> >> handling of the above exception, another exception occurred:
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server Traceback
> >> (most recent call last):
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server   File
> >> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in
> >> _process_incoming
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server     res =
> >> self.dispatcher.dispatch(message)
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server   File
> >> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in
> >> dispatch
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
> >> return self._do_dispatch(endpoint, method, ctxt, args)
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server   File
> >> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in
> >> _do_dispatch
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
> >> result = func(ctxt, **new_args)
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server   File
> >> "/var/lib/kolla/venv/lib/python3.6/site-packages/manila/share/manager.py",
> >> line 187, in wrapped
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
> >> return f(self, *args, **kwargs)
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server   File
> >> "/var/lib/kolla/venv/lib/python3.6/site-packages/manila/utils.py",
> >> line 568, in wrapper
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
> >> return func(self, *args, **kwargs)
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server   File
> >> "/var/lib/kolla/venv/lib/python3.6/site-packages/manila/share/manager.py",
> >> line 1790, in create_share_instance
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server     exception=e)
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server   File
> >> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> >> line 220, in __exit__
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
> >> self.force_reraise()
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server   File
> >> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> >> line 196, in force_reraise
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
> >> six.reraise(self.type_, self.value, self.tb)
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server   File
> >> "/var/lib/kolla/venv/lib/python3.6/site-packages/six.py", line 693,
> >> in reraise
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server     raise value
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server   File
> >> "/var/lib/kolla/venv/lib/python3.6/site-packages/manila/share/manager.py",
> >> line 1753, in create_share_instance
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
> >> context, share_instance, share_server=share_server)
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server   File
> >> "/var/lib/kolla/venv/lib/python3.6/site-packages/manila/share/drivers/cephfs/driver.py", line 272, in
> >> create_share
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
> >> mode=self._cephfs_volume_mode)
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server   File
> >> "/usr/lib/python3/dist-packages/ceph_volume_client.py", line 677, in
> >> create_volume
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
> >> self._mkdir_p(path, mode)
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server   File
> >> "/usr/lib/python3/dist-packages/ceph_volume_client.py", line 658, in
> >> _mkdir_p
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
> >> self.fs.mkdir(subpath, mode)
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server   File
> >> "cephfs.pyx", line 887, in cephfs.LibCephFS.mkdir
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
> >> cephfs.OSError: error in mkdir
> >> volumes/_nogroup/449a52a4-f19c-4d1b-b437-7dc2443e040c: Permission
> >> denied [Errno 13]
> >> 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server
> >
> >
> >
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220216/48267cc5/attachment-0001.htm>