[kolla-ansible][yoga] Glance backend cinder Privsep daemon failed to start operation not permitted
Rajat Dhasmana
rdhasman at redhat.com
Tue May 30 11:02:10 UTC 2023
Hi,
On Mon, Apr 17, 2023 at 9:19 PM wodel youchi <wodel.youchi at gmail.com> wrote:
> Hi,
>
> I managed to get another step forward, but I hit another wall.
> Glance-api was trying to mount the NFS share on the controller node, not
> on the compute, so I installed nfs-utils on it, now I have this error :
>
>
Glance will mount the share on the node in which it's running. The default
location for mounting is "/var/lib/glance/mnt" and can be configured via
config option "cinder_mount_point_base"[1]
Since Glance is running on the controller node in your case, it will mount
the share on the controller.
> 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder
>> [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 - default default] Exception while
>> accessing to cinder volume 3d4712ef-7cb5-4a0b-bc1e-cfbebd8fa902.:
>> FileExistsError: [Errno 17] File exists:
>> '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7'
>> 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder Traceback
>> (most recent call last):
>> 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py",
>> line 788, in _open_cinder_volume
>> 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder device
>> = connect_volume_nfs()
>> 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py",
>> line 391, in inner
>> 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder return
>> f(*args, **kwargs)
>> 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py",
>> line 786, in connect_volume_nfs
>> 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder
>> root_helper, options)
>> 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py",
>> line 359, in mount
>> 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder
>> rootwrap_helper, options)
>> 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py",
>> line 247, in mount
>> 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder
>> os.makedirs(mountpoint)
>> 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File
>> "/usr/lib64/python3.6/os.py", line 220, in makedirs
>> 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder
>> mkdir(name, mode)
>> 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder
>> FileExistsError: [Errno 17] File exists:
>> '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7'
>> 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder
>> 2023-04-17 16:33:08.906 62 ERROR glance_store._drivers.cinder
>> [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 - default default] Failed to write to
>> volume 3d4712ef-7cb5-4a0b-bc1e-cfbebd8fa902.: FileExistsError: [Errno 17]
>> File exists:
>> '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7'
>> 2023-04-17 16:33:08.939 62 ERROR glance.api.v2.image_data
>> [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 - default default] Failed to upload image
>> data due to internal error: FileExistsError: [Errno 17] File exists:
>> '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7'
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi
>> [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 - default default] Caught error: [Errno
>> 17] File exists:
>> '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7':
>> FileExistsError: [Errno 17] File exists:
>> '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7'
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi Traceback (most
>> recent call last):
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py",
>> line 1332, in __call__
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi request,
>> **action_args)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py",
>> line 1370, in dispatch
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return
>> method(*args, **kwargs)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/utils.py",
>> line 414, in wrapped
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return func(self,
>> req, *args, **kwargs)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py",
>> line 303, in upload
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi
>> self._restore(image_repo, image)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 227, in __exit__
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi
>> self.force_reraise()
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 200, in force_reraise
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise self.value
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py",
>> line 163, in upload
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi
>> image.set_data(data, size, backend=backend)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line
>> 497, in set_data
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi
>> _send_notification(notify_error, 'image.upload', msg)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 227, in __exit__
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi
>> self.force_reraise()
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 200, in force_reraise
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise self.value
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line
>> 444, in set_data
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi
>> set_active=set_active)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/quota/__init__.py",
>> line 323, in set_data
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi
>> set_active=set_active)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line
>> 585, in set_data
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi
>> self._upload_to_store(data, verifier, backend, size)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line
>> 485, in _upload_to_store
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi verifier=verifier)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py",
>> line 399, in add_with_multihash
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image_id, data,
>> size, hashing_algo, store, context, verifier)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py",
>> line 481, in store_add_to_backend_with_multihash
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image_id, data,
>> size, hashing_algo, context=context, verifier=verifier)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/driver.py",
>> line 279, in add_adapter
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi metadata_dict) =
>> store_add_fun(*args, **kwargs)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/capabilities.py",
>> line 176, in op_checker
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return
>> store_op_fun(store, *args, **kwargs)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py",
>> line 1028, in add
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise
>> errors.get(e.errno, e)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py",
>> line 985, in add
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi with
>> self._open_cinder_volume(client, volume, 'wb') as f:
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return
>> next(self.gen)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py",
>> line 788, in _open_cinder_volume
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi device =
>> connect_volume_nfs()
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py",
>> line 391, in inner
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return f(*args,
>> **kwargs)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py",
>> line 786, in connect_volume_nfs
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi root_helper,
>> options)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py",
>> line 359, in mount
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi rootwrap_helper,
>> options)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py",
>> line 247, in mount
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi
>> os.makedirs(mountpoint)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File
>> "/usr/lib64/python3.6/os.py", line 220, in makedirs
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi mkdir(name, mode)
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi FileExistsError:
>> [Errno 17] File exists:
>> '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7'
>> 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi
>>
>
>
This is a strange error, we acquire a lock on the export path before it is
mounted[2] and again acquire a threading lock on the mountpoint[3].
Then we check if the path is already mounted and only create the directory.
we also remove the directory during unmount[4].
Are you using multiple glance processes with concurrent operations running?
I can see we have a limitation because of the threading lock
that only prevents access to the thread of the same process.
[1]
https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc19744e404015/glance_store/_drivers/cinder/store.py#L399
[2]
https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc19744e404015/glance_store/_drivers/cinder/nfs.py#L82
[3]
https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc19744e404015/glance_store/common/fs_mount.py#L242
[4]
https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc19744e404015/glance_store/common/fs_mount.py#L339
Thanks
Rajat Dhasmana
> Regards.
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> Virus-free.www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_-7349147206883296287_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230530/43d55b48/attachment-0001.htm>
More information about the openstack-discuss
mailing list