<div dir="ltr"><div>Hi,</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 17, 2023 at 9:19 PM wodel youchi <<a href="mailto:wodel.youchi@gmail.com">wodel.youchi@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>Hi,</div><div><br></div><div>I managed to get another step forward, but I hit another wall.</div><div>Glance-api was trying to mount the NFS share on the controller node, not on the compute, so I installed nfs-utils on it, now I have this error :</div><div><br></div></div></div></blockquote><div><br></div><div>Glance will mount the share on the node in which it's running. The default location for mounting is "/var/lib/glance/mnt" and can be configured via config option "cinder_mount_point_base"[1]</div><div>Since Glance is running on the controller node in your case, it will mount the share on the controller.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Exception while accessing to cinder volume 3d4712ef-7cb5-4a0b-bc1e-cfbebd8fa902.: FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7'<br>2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder Traceback (most recent call last):<br>2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 788, in _open_cinder_volume<br>2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder device = connect_volume_nfs()<br>2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 391, in inner<br>2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder return f(*args, **kwargs)<br>2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 786, in connect_volume_nfs<br>2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder root_helper, options)<br>2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 359, in mount<br>2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder rootwrap_helper, options)<br>2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 247, in mount<br>2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder os.makedirs(mountpoint)<br>2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/usr/lib64/python3.6/os.py", line 220, in makedirs<br>2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder mkdir(name, mode)<br>2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7'<br>2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder<br>2023-04-17 16:33:08.906 62 ERROR glance_store._drivers.cinder [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Failed to write to volume 3d4712ef-7cb5-4a0b-bc1e-cfbebd8fa902.: FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7'<br>2023-04-17 16:33:08.939 62 ERROR glance.api.v2.image_data [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Failed to upload image data due to internal error: FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7'<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Caught error: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7': FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7'<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi Traceback (most recent call last):<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1332, in __call__<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi request, **action_args)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1370, in dispatch<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return method(*args, **kwargs)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/utils.py", line 414, in wrapped<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 303, in upload<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self._restore(image_repo, image)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self.force_reraise()<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise self.value<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 163, in upload<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image.set_data(data, size, backend=backend)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 497, in set_data<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self.force_reraise()<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise self.value<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 444, in set_data<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi set_active=set_active)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/quota/__init__.py", line 323, in set_data<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi set_active=set_active)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 585, in set_data<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 485, in _upload_to_store<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi verifier=verifier)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 399, in add_with_multihash<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 481, in store_add_to_backend_with_multihash<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/driver.py", line 279, in add_adapter<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/capabilities.py", line 176, in op_checker<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 1028, in add<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise errors.get(e.errno, e)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 985, in add<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi with self._open_cinder_volume(client, volume, 'wb') as f:<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return next(self.gen)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 788, in _open_cinder_volume<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi device = connect_volume_nfs()<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 391, in inner<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return f(*args, **kwargs)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 786, in connect_volume_nfs<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi root_helper, options)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 359, in mount<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi rootwrap_helper, options)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 247, in mount<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi os.makedirs(mountpoint)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/usr/lib64/python3.6/os.py", line 220, in makedirs<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi mkdir(name, mode)<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7'<br>2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi</div></blockquote></div><div class="gmail_quote"><br></div></div></blockquote><div><br></div><div>This is a strange error, we acquire a lock on the export path before it is mounted[2] and again acquire a threading lock on the mountpoint[3].</div><div>Then we check if the path is already mounted and only create the directory. we also remove the directory during unmount[4].</div><div>Are you using multiple glance processes with concurrent operations running? I can see we have a limitation because of the threading lock</div><div>that only prevents access to the thread of the same process.</div><div><br></div><div><div>[1] <a href="https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc19744e404015/glance_store/_drivers/cinder/store.py#L399">https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc19744e404015/glance_store/_drivers/cinder/store.py#L399</a></div></div><div>[2] <a href="https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc19744e404015/glance_store/_drivers/cinder/nfs.py#L82">https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc19744e404015/glance_store/_drivers/cinder/nfs.py#L82</a></div><div>[3] <a href="https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc19744e404015/glance_store/common/fs_mount.py#L242">https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc19744e404015/glance_store/common/fs_mount.py#L242</a></div><div>[4] <a href="https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc19744e404015/glance_store/common/fs_mount.py#L339">https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc19744e404015/glance_store/common/fs_mount.py#L339</a></div><div> </div><div>Thanks</div><div>Rajat Dhasmana</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"></div><div class="gmail_quote"><br></div><div class="gmail_quote">Regards.<br></div></div><div id="m_-7349147206883296287DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2"><br><table style="border-top:1px solid rgb(211,212,222)"><tbody><tr><td style="width:55px;padding-top:13px"><a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail" target="_blank"><img src="https://s-install.avcdn.net/ipm/preview/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif" alt="" width="46" height="29" style="width: 46px; height: 29px;"></a></td><td style="width:470px;padding-top:12px;color:rgb(65,66,78);font-size:13px;font-family:Arial,Helvetica,sans-serif;line-height:18px">Virus-free.<a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail" style="color:rgb(68,83,234)" target="_blank">www.avast.com</a></td></tr></tbody></table><a href="#m_-7349147206883296287_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2" width="1" height="1"></a></div>
</blockquote></div></div>