On Mon, Apr 17, 2023 at 9:19 PM wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,I managed to get another step forward, but I hit another wall.Glance-api was trying to mount the NFS share on the controller node, not on the compute, so I installed nfs-utils on it, now I have this error :
Glance will mount the share on the node in which it's running. The default location for mounting is "/var/lib/glance/mnt" and can be configured via config option "cinder_mount_point_base"[1]
Since Glance is running on the controller node in your case, it will mount the share on the controller.
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Exception while accessing to cinder volume 3d4712ef-7cb5-4a0b-bc1e-cfbebd8fa902.: FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7'
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder Traceback (most recent call last):
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 788, in _open_cinder_volume
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder device = connect_volume_nfs()
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 391, in inner
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder return f(*args, **kwargs)
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 786, in connect_volume_nfs
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder root_helper, options)
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 359, in mount
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder rootwrap_helper, options)
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 247, in mount
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder os.makedirs(mountpoint)
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/usr/lib64/python3.6/os.py", line 220, in makedirs
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder mkdir(name, mode)
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7'
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder
2023-04-17 16:33:08.906 62 ERROR glance_store._drivers.cinder [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Failed to write to volume 3d4712ef-7cb5-4a0b-bc1e-cfbebd8fa902.: FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7'
2023-04-17 16:33:08.939 62 ERROR glance.api.v2.image_data [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Failed to upload image data due to internal error: FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7'
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Caught error: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7': FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7'
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi Traceback (most recent call last):
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1332, in __call__
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi request, **action_args)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1370, in dispatch
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return method(*args, **kwargs)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/utils.py", line 414, in wrapped
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 303, in upload
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self._restore(image_repo, image)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self.force_reraise()
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise self.value
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 163, in upload
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image.set_data(data, size, backend=backend)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 497, in set_data
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self.force_reraise()
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise self.value
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 444, in set_data
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi set_active=set_active)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/quota/__init__.py", line 323, in set_data
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi set_active=set_active)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 585, in set_data
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 485, in _upload_to_store
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi verifier=verifier)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 399, in add_with_multihash
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 481, in store_add_to_backend_with_multihash
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/driver.py", line 279, in add_adapter
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/capabilities.py", line 176, in op_checker
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 1028, in add
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise errors.get(e.errno, e)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 985, in add
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi with self._open_cinder_volume(client, volume, 'wb') as f:
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return next(self.gen)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 788, in _open_cinder_volume
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi device = connect_volume_nfs()
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 391, in inner
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return f(*args, **kwargs)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 786, in connect_volume_nfs
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi root_helper, options)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 359, in mount
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi rootwrap_helper, options)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 247, in mount
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi os.makedirs(mountpoint)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/usr/lib64/python3.6/os.py", line 220, in makedirs
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi mkdir(name, mode)
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7'
2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi
This is a strange error, we acquire a lock on the export path before it is mounted[2] and again acquire a threading lock on the mountpoint[3].
Then we check if the path is already mounted and only create the directory. we also remove the directory during unmount[4].
Are you using multiple glance processes with concurrent operations running? I can see we have a limitation because of the threading lock
that only prevents access to the thread of the same process.
Thanks
Rajat Dhasmana
Regards.
