Hi, On Mon, Apr 17, 2023 at 9:19 PM wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
I managed to get another step forward, but I hit another wall. Glance-api was trying to mount the NFS share on the controller node, not on the compute, so I installed nfs-utils on it, now I have this error :
Glance will mount the share on the node in which it's running. The default location for mounting is "/var/lib/glance/mnt" and can be configured via config option "cinder_mount_point_base"[1] Since Glance is running on the controller node in your case, it will mount the share on the controller.
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder
[req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Exception while accessing to cinder volume 3d4712ef-7cb5-4a0b-bc1e-cfbebd8fa902.: FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder Traceback (most recent call last): 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 788, in _open_cinder_volume 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder device = connect_volume_nfs() 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 391, in inner 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder return f(*args, **kwargs) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 786, in connect_volume_nfs 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder root_helper, options) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 359, in mount 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder rootwrap_helper, options) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 247, in mount 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder os.makedirs(mountpoint) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/usr/lib64/python3.6/os.py", line 220, in makedirs 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder mkdir(name, mode) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder 2023-04-17 16:33:08.906 62 ERROR glance_store._drivers.cinder [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Failed to write to volume 3d4712ef-7cb5-4a0b-bc1e-cfbebd8fa902.: FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.939 62 ERROR glance.api.v2.image_data [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Failed to upload image data due to internal error: FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Caught error: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7': FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi Traceback (most recent call last): 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1332, in __call__ 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi request, **action_args) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1370, in dispatch 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return method(*args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/utils.py", line 414, in wrapped 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 303, in upload 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self._restore(image_repo, image) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self.force_reraise() 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise self.value 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 163, in upload 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 497, in set_data 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self.force_reraise() 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise self.value 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 444, in set_data 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi set_active=set_active) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/quota/__init__.py", line 323, in set_data 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi set_active=set_active) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 585, in set_data 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 485, in _upload_to_store 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi verifier=verifier) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 399, in add_with_multihash 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 481, in store_add_to_backend_with_multihash 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/driver.py", line 279, in add_adapter 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/capabilities.py", line 176, in op_checker 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 1028, in add 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise errors.get(e.errno, e) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 985, in add 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi with self._open_cinder_volume(client, volume, 'wb') as f: 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return next(self.gen) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 788, in _open_cinder_volume 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi device = connect_volume_nfs() 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 391, in inner 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return f(*args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 786, in connect_volume_nfs 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi root_helper, options) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 359, in mount 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi rootwrap_helper, options) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 247, in mount 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi os.makedirs(mountpoint) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/usr/lib64/python3.6/os.py", line 220, in makedirs 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi mkdir(name, mode) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi
This is a strange error, we acquire a lock on the export path before it is mounted[2] and again acquire a threading lock on the mountpoint[3]. Then we check if the path is already mounted and only create the directory. we also remove the directory during unmount[4]. Are you using multiple glance processes with concurrent operations running? I can see we have a limitation because of the threading lock that only prevents access to the thread of the same process. [1] https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc1... [2] https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc1... [3] https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc1... [4] https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc1... Thanks Rajat Dhasmana
Regards.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#m_-7349147206883296287_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>