[kolla-ansible][yoga] Glance backend cinder Privsep daemon failed to start operation not permitted
Hi,
I am trying to configure glance to use cinder as a backend.
This is my glance-api.conf [cinder] cinder_store_auth_address = https://dashint.example.com:5000/v3 cinder_store_user_name = cinder cinder_store_password = cinderpass cinder_store_project_name = service cinder_volume_type = nfstype rootwrap_config = /etc/glance/rootwrap.conf
==> /var/log/kolla/glance/glance-api.log <==
2023-04-12 18:02:20.842 64 INFO oslo.privsep.daemon [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Spawned new privsep daemon via rootwrap 2023-04-12 18:02:20.733 360 INFO oslo.privsep.daemon [-] privsep daemon starting 2023-04-12 18:02:20.735 360 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
*2023-04-12 18:02:20.737 360 ERROR oslo.privsep.daemon [-] [Errno 1] Operation not permitted Traceback (most recent call last): * File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 584, in helper_main Daemon(channel, context).run() File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 394, in run self._drop_privs() File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 430, in _drop_privs capabilities.drop_all_caps_except(self.caps, self.caps, []) File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/capabilities.py", line 156, in drop_all_caps_except raise OSError(errno, os.strerror(errno)) PermissionError: [Errno 1] Operation not permitted 2023-04-12 18:02:20.844 64 WARNING oslo_privsep.comm [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Unexpected error: <class 'BrokenPipeError'>: BrokenPipeError: [Errno 32] Broken pipe 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Error while sending initial PING to privsep: [Errno 32] Broken pipe: BrokenPipeError: [Errno 32] Broken pipe 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon Traceback (most recent call last): 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 192, in exchange_ping 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon reply = self.send_recv((comm.Message.PING.value,)) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/comm.py", line 186, in send_recv 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon self.writer.send((myid, msg)) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/comm.py", line 60, in send 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon self.writesock.sendall(buf) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py", line 407, in sendall 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon tail = self.send(data, flags) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py", line 401, in send 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon return self._send_loop(self.fd.send, data, flags) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py", line 388, in _send_loop 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon return send_method(data, *args) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon BrokenPipeError: [Errno 32] Broken pipe 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon 2023-04-12 18:02:20.846 64 CRITICAL oslo.privsep.daemon [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - defau lt default] Privsep daemon failed to start 2023-04-12 18:02:20.847 64 ERROR glance_store._drivers.cinder [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Failed to write to volume 46316c12-6c24-40af-afde-1c16edd616b6.: oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start 2023-04-12 18:02:20.890 64 ERROR glance.api.v2.image_data [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - def ault default] *Failed to upload image data due to internal error: oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start * 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default d efault] Caught error: Privsep daemon failed to start: oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi Traceback (most recent call last): 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1332, in __call__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi request, **action_args) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1370, in dispatch 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return method(*args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/utils.py", line 414, in wrapped 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 303, in upload 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self._restore(image_repo, image) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.force_reraise() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise self.value 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 163, in upload 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 497, in set_data 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.force_reraise() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise self.value 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 444, in set_data 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi set_active=set_active) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/quota/__init__.py", line 323, in set_data 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi set_active=set_active) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 585, in set_data 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 485, in _upload_to_store 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi verifier=verifier) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 399, in add_with_multihash 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 481, in store_add_to_backe nd_with_multihash 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/driver.py", line 279, in add_adapter 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/capabilities.py", line 176, in op_checker 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 985, in add 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi with self._open_cinder_volume(client, volume, 'wb') as f: 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return next(self.gen) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 739, in _open_cinder_vol ume 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi root_helper, my_ip, use_multipath, enforce_multipath, host=host) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/utils.py", line 169, in trace_logging_wrapper 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return f(*args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connector.py", line 240, in get_connector_pr operties 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi execute=execute)) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 70, in get_connec tor_properties 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi initiator = iscsi.get_initiator() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 963, in get_initi ator 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi root_helper=self._root_helper) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/executor.py", line 53, in _execute 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi result = self.__execute(*args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 172, in execute 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return execute_root(*cmd, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 269, in _wrap 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.start() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 283, in start 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi channel = daemon.RootwrapClientChannel(context=self) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 374, in __init__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi super(RootwrapClientChannel, self).__init__(sock, context) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 187, in __init__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.exchange_ping() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 201, in exchange_ping 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise FailedToDropPrivileges(msg) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi *oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start * 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi 2023-04-12 18:02:20.927 64 INFO eventlet.wsgi.server [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] 20.3.0.34,127.0.0.1 - - [12/Apr/2023 18:02:20] "PUT /v2/images/52ed7ed7-330e-4249-abb9-5ec99712846f/file HTTP/1.1" 500 430 2.727683
It seems like a lack of privileges, any ideas?
Regards.
On Wed, Apr 12, 2023 at 10:41 AM wodel youchi wodel.youchi@gmail.com wrote:
Hi,
I am trying to configure glance to use cinder as a backend.
This is my glance-api.conf [cinder] cinder_store_auth_address = https://dashint.example.com:5000/v3 cinder_store_user_name = cinder cinder_store_password = cinderpass cinder_store_project_name = service cinder_volume_type = nfstype rootwrap_config = /etc/glance/rootwrap.conf
==> /var/log/kolla/glance/glance-api.log <==
2023-04-12 18:02:20.842 64 INFO oslo.privsep.daemon [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Spawned new privsep daemon via rootwrap 2023-04-12 18:02:20.733 360 INFO oslo.privsep.daemon [-] privsep daemon starting 2023-04-12 18:02:20.735 360 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
*2023-04-12 18:02:20.737 360 ERROR oslo.privsep.daemon [-] [Errno 1] Operation not permitted Traceback (most recent call last): * File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 584, in helper_main Daemon(channel, context).run() File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 394, in run self._drop_privs() File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 430, in _drop_privs capabilities.drop_all_caps_except(self.caps, self.caps, []) File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/capabilities.py", line 156, in drop_all_caps_except raise OSError(errno, os.strerror(errno)) PermissionError: [Errno 1] Operation not permitted 2023-04-12 18:02:20.844 64 WARNING oslo_privsep.comm [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Unexpected error: <class 'BrokenPipeError'>: BrokenPipeError: [Errno 32] Broken pipe 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Error while sending initial PING to privsep: [Errno 32] Broken pipe: BrokenPipeError: [Errno 32] Broken pipe 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon Traceback (most recent call last): 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 192, in exchange_ping 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon reply = self.send_recv((comm.Message.PING.value,)) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/comm.py", line 186, in send_recv 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon self.writer.send((myid, msg)) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/comm.py", line 60, in send 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon self.writesock.sendall(buf) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py", line 407, in sendall 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon tail = self.send(data, flags) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py", line 401, in send 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon return self._send_loop(self.fd.send, data, flags) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py", line 388, in _send_loop 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon return send_method(data, *args) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon BrokenPipeError: [Errno 32] Broken pipe 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon 2023-04-12 18:02:20.846 64 CRITICAL oslo.privsep.daemon [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - defau lt default] Privsep daemon failed to start 2023-04-12 18:02:20.847 64 ERROR glance_store._drivers.cinder [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Failed to write to volume 46316c12-6c24-40af-afde-1c16edd616b6.: oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start 2023-04-12 18:02:20.890 64 ERROR glance.api.v2.image_data [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - def ault default] *Failed to upload image data due to internal error: oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start * 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default d efault] Caught error: Privsep daemon failed to start: oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi Traceback (most recent call last): 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1332, in __call__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi request, **action_args) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1370, in dispatch 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return method(*args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/utils.py", line 414, in wrapped 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 303, in upload 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self._restore(image_repo, image) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.force_reraise() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise self.value 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 163, in upload 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 497, in set_data 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.force_reraise() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise self.value 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 444, in set_data 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi set_active=set_active) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/quota/__init__.py", line 323, in set_data 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi set_active=set_active) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 585, in set_data 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 485, in _upload_to_store 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi verifier=verifier) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 399, in add_with_multihash 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 481, in store_add_to_backe nd_with_multihash 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/driver.py", line 279, in add_adapter 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/capabilities.py", line 176, in op_checker 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 985, in add 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi with self._open_cinder_volume(client, volume, 'wb') as f: 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return next(self.gen) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 739, in _open_cinder_vol ume 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi root_helper, my_ip, use_multipath, enforce_multipath, host=host) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/utils.py", line 169, in trace_logging_wrapper 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return f(*args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connector.py", line 240, in get_connector_pr operties 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi execute=execute)) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 70, in get_connec tor_properties 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi initiator = iscsi.get_initiator() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 963, in get_initi ator 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi root_helper=self._root_helper) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/executor.py", line 53, in _execute 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi result = self.__execute(*args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 172, in execute 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return execute_root(*cmd, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 269, in _wrap 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.start() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 283, in start 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi channel = daemon.RootwrapClientChannel(context=self) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 374, in __init__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi super(RootwrapClientChannel, self).__init__(sock, context) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 187, in __init__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.exchange_ping() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 201, in exchange_ping 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise FailedToDropPrivileges(msg) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi *oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start * 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi 2023-04-12 18:02:20.927 64 INFO eventlet.wsgi.server [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] 20.3.0.34,127.0.0.1 - - [12/Apr/2023 18:02:20] "PUT /v2/images/52ed7ed7-330e-4249-abb9-5ec99712846f/file HTTP/1.1" 500 430 2.727683
It seems like a lack of privileges, any ideas?
Yes, the glance-api container itself must run with "privileged: true" when glance is using cinder for a backend. For reference, you can see how TripleO handles this:
https://github.com/openstack/tripleo-heat-templates/blob/2e6d826debd6099b3d8...
Alan
Regards.
On Wed, 2023-04-12 at 11:12 -0700, Alan Bishop wrote:
On Wed, Apr 12, 2023 at 10:41 AM wodel youchi wodel.youchi@gmail.com wrote:
Hi,
I am trying to configure glance to use cinder as a backend.
This is my glance-api.conf [cinder] cinder_store_auth_address = https://dashint.example.com:5000/v3 cinder_store_user_name = cinder cinder_store_password = cinderpass cinder_store_project_name = service cinder_volume_type = nfstype rootwrap_config = /etc/glance/rootwrap.conf
==> /var/log/kolla/glance/glance-api.log <==
2023-04-12 18:02:20.842 64 INFO oslo.privsep.daemon [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Spawned new privsep daemon via rootwrap 2023-04-12 18:02:20.733 360 INFO oslo.privsep.daemon [-] privsep daemon starting 2023-04-12 18:02:20.735 360 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
*2023-04-12 18:02:20.737 360 ERROR oslo.privsep.daemon [-] [Errno 1] Operation not permitted Traceback (most recent call last): * File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 584, in helper_main Daemon(channel, context).run() File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 394, in run self._drop_privs() File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 430, in _drop_privs capabilities.drop_all_caps_except(self.caps, self.caps, []) File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/capabilities.py", line 156, in drop_all_caps_except raise OSError(errno, os.strerror(errno)) PermissionError: [Errno 1] Operation not permitted 2023-04-12 18:02:20.844 64 WARNING oslo_privsep.comm [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Unexpected error: <class 'BrokenPipeError'>: BrokenPipeError: [Errno 32] Broken pipe 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Error while sending initial PING to privsep: [Errno 32] Broken pipe: BrokenPipeError: [Errno 32] Broken pipe 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon Traceback (most recent call last): 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 192, in exchange_ping 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon reply = self.send_recv((comm.Message.PING.value,)) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/comm.py", line 186, in send_recv 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon self.writer.send((myid, msg)) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/comm.py", line 60, in send 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon self.writesock.sendall(buf) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py", line 407, in sendall 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon tail = self.send(data, flags) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py", line 401, in send 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon return self._send_loop(self.fd.send, data, flags) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py", line 388, in _send_loop 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon return send_method(data, *args) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon BrokenPipeError: [Errno 32] Broken pipe 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon 2023-04-12 18:02:20.846 64 CRITICAL oslo.privsep.daemon [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - defau lt default] Privsep daemon failed to start 2023-04-12 18:02:20.847 64 ERROR glance_store._drivers.cinder [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Failed to write to volume 46316c12-6c24-40af-afde-1c16edd616b6.: oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start 2023-04-12 18:02:20.890 64 ERROR glance.api.v2.image_data [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - def ault default] *Failed to upload image data due to internal error: oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start * 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default d efault] Caught error: Privsep daemon failed to start: oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi Traceback (most recent call last): 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1332, in __call__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi request, **action_args) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1370, in dispatch 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return method(*args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/utils.py", line 414, in wrapped 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 303, in upload 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self._restore(image_repo, image) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.force_reraise() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise self.value 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 163, in upload 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 497, in set_data 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.force_reraise() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise self.value 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 444, in set_data 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi set_active=set_active) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/quota/__init__.py", line 323, in set_data 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi set_active=set_active) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 585, in set_data 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 485, in _upload_to_store 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi verifier=verifier) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 399, in add_with_multihash 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 481, in store_add_to_backe nd_with_multihash 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/driver.py", line 279, in add_adapter 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/capabilities.py", line 176, in op_checker 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 985, in add 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi with self._open_cinder_volume(client, volume, 'wb') as f: 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return next(self.gen) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 739, in _open_cinder_vol ume 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi root_helper, my_ip, use_multipath, enforce_multipath, host=host) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/utils.py", line 169, in trace_logging_wrapper 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return f(*args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connector.py", line 240, in get_connector_pr operties 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi execute=execute)) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 70, in get_connec tor_properties 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi initiator = iscsi.get_initiator() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 963, in get_initi ator 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi root_helper=self._root_helper) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/executor.py", line 53, in _execute 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi result = self.__execute(*args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 172, in execute 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return execute_root(*cmd, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 269, in _wrap 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.start() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 283, in start 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi channel = daemon.RootwrapClientChannel(context=self) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 374, in __init__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi super(RootwrapClientChannel, self).__init__(sock, context) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 187, in __init__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.exchange_ping() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 201, in exchange_ping 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise FailedToDropPrivileges(msg) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi *oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start * 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi 2023-04-12 18:02:20.927 64 INFO eventlet.wsgi.server [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] 20.3.0.34,127.0.0.1 - - [12/Apr/2023 18:02:20] "PUT /v2/images/52ed7ed7-330e-4249-abb9-5ec99712846f/file HTTP/1.1" 500 430 2.727683
It seems like a lack of privileges, any ideas?
Yes, the glance-api container itself must run with "privileged: true" when glance is using cinder for a backend. For reference, you can see how TripleO handles this:
https://github.com/openstack/tripleo-heat-templates/blob/2e6d826debd6099b3d8...
that should work in yoga i have my home cloud using glance-api with cinder back storage
sean@cloud:~/repos/kolla-ansible$ cat /etc/kolla/config/glance/glance-api.conf [DEFAULT] enabled_import_methods = ['glance-direct','web-download','copy-image'] enabled_backends = local:cinder cinder_catalog_info = volumev3:cinderv3:internalURL show_multiple_locations = True show_image_direct_url = True
[glance_store] default_backend = local default_store = cinder stores = cinder
[keystone_authtoken] interface = internal
[local] cinder_volume_type = local_storage description = LVM based cinder store cinder_catalog_info = volumev3:cinderv3:internalURL cinder_store_auth_address = {{ keystone_internal_url }} cinder_store_project_name = service cinder_store_user_name = {{ glance_keystone_user }} cinder_store_password = {{ glance_keystone_password }}
my glance options in gobal.yamlare
######################## # Glance - Image Options ######################## # Configure image backend. #glance_backend_ceph: "no" glance_backend_file: "no" #glance_backend_swift: "no" #glance_backend_vmware: "no" #enable_glance_image_cache: "no" glance_enable_property_protection: "no" glance_enable_interoperable_image_import: "yes" # Configure glance upgrade option. # Due to this feature being experimental in glance, # the default value is "no". #glance_enable_rolling_upgrade: "no"
and cinder lvm
# Cinder - Block Storage Options ################################ # Enable / disable Cinder backends #cinder_backend_ceph: "no" #cinder_backend_vmwarevc_vmdk: "no" #cinder_backend_vmware_vstorage_object: "no" cinder_volume_group: "cinder-volumes" # Valid options are [ '', redis, etcd ] cinder_coordination_backend: ""
on the cidner side i have sean@cloud:~/repos/kolla-ansible$ cat /etc/kolla/config/cinder.conf [DEFAULT] allowed_direct_url_schemes = cinder [lvm-1] image_upload_use_cinder_backend = True image_upload_use_internal_tenant = True
otherwise i requried no other change in kolla-ansible stable/yoga
commit bf1e29c87b52cd4a75f484c1f16c265fdb09b5fa (HEAD -> stable/yoga, origin/stable/yoga) Merge: 4ee8667f1 f4e6dcc3b Author: Zuul zuul@review.opendev.org Date: Tue Sep 20 10:00:54 2022 +0000
Merge "Fix prometheus-msteams image name" into stable/yoga
perhaps the nfs type has extra requiement btu lvm should work.
Alan
Regards.
Thanks for the help,
I modified my glance-api.conf like this but no luck, I got the same problem : Privsep daemon failed to start
[cinder] cinder_store_auth_address = https://dashint.example.com:5000/v3 cinder_store_project_name = service cinder_volume_type = nfstype rootwrap_config = /etc/glance/rootwrap.conf
*cinder_store_user_name = glancecinder_store_password = glance-password* cinder_catalog_info = volumev3:cinderv3:internalURL
*###cinder_store_user_name = cinder###cinder_store_password = cinder-password *
I thought once cinder has the capability to read & write into the backend, glance will just use it to upload the images!!
Another thing, about the credentials, which ones should I use? Glance's or cinder's I thought it should be cinder's !! but in your example @Sean you used {{ glance_keystone_user }} and {{ glance_keystone_password }}
Lastly I searched how to launch a privileged container in kolla but I didn't find anything.
Regards.
On Thu, 2023-04-13 at 11:41 +0100, wodel youchi wrote:
Thanks for the help,
I modified my glance-api.conf like this but no luck, I got the same problem : Privsep daemon failed to start
[cinder] cinder_store_auth_address = https://dashint.example.com:5000/v3 cinder_store_project_name = service cinder_volume_type = nfstype rootwrap_config = /etc/glance/rootwrap.conf
*cinder_store_user_name = glancecinder_store_password = glance-password* cinder_catalog_info = volumev3:cinderv3:internalURL
*###cinder_store_user_name = cinder###cinder_store_password = cinder-password *
I thought once cinder has the capability to read & write into the backend, glance will just use it to upload the images!!
Another thing, about the credentials, which ones should I use? Glance's or cinder's I thought it should be cinder's !! but in your example @Sean you used {{ glance_keystone_user }} and {{ glance_keystone_password }}
either would work but normally i woudl consider it bad partice for a service to have the username/password for a differnt users account so glance should alwasy use its user to talk to other services when its not using the keystone user token passed in to the api.
Lastly I searched how to launch a privileged container in kolla but I didn't find anything.
i see the issue https://github.com/openstack/kolla-ansible/blob/stable/yoga/ansible/roles/gl... glace api is only privlaged if you use iscsi for the cinder backend. privileged: "{{ enable_cinder | bool and enable_cinder_backend_iscsi | bool }}" you are using nfs.
you can workaround this in your global.yaml by adding this i think
glance_services: glance-api: privileged: true
alternitivly you can do it via the inventory file by seting that as a host/group var.
Regards.
Hi,
Adding those lines to globals.yml did produce errors, complaining about the enabled line missing
glance_services: glance-api: privileged: true
Then I tried this
glance_services: glance-api: enabled: true privileged: true
Gave these errors fatal: [192.168.2.23]: FAILED! => {"msg": "The conditional check 'item.value.host_in_groups | bool' failed. The error was: error while evaluating conditional (item.value.host_in_groups | bool): 'dict object' has no attribute 'host_in_groups'\n\nThe error appears to be in '/home/deployer/kollavenv/yogavenv/share/kolla-ansible/ansible/roles/glance/tasks/config.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Ensuring config directories exist\n ^ here\n"} fatal: [192.168.2.27]: FAILED! => {"msg": "The conditional check 'item.value.host_in_groups | bool' failed. The error was: error while evaluating conditional (item.value.host_in_groups | bool): 'dict object' has no attribute 'host_in_groups'\n\nThe error appears to be in '/home/deployer/kollavenv/yogavenv/share/kolla-ansible/ansible/roles/glance/tasks/config.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Ensuring config directories exist\n ^ here\n"} fatal: [192.168.2.31]: FAILED! => {"msg": "The conditional check 'item.value.host_in_groups | bool' failed. The error was: error while evaluating conditional (item.value.host_in_groups | bool): 'dict object' has no attribute 'host_in_groups'\n\nThe error appears to be in '/home/deployer/kollavenv/yogavenv/share/kolla-ansible/ansible/roles/glance/tasks/config.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Ensuring config directories exist\n ^ here\n"}
it seems, this configuration is discarding main.yml from https://github.com/openstack/kolla-ansible/blob/stable/yoga/ansible/roles/gl... not just replacing the needed variables.
Regards.
Le jeu. 13 avr. 2023 à 13:00, Sean Mooney smooney@redhat.com a écrit :
On Thu, 2023-04-13 at 11:41 +0100, wodel youchi wrote:
Thanks for the help,
I modified my glance-api.conf like this but no luck, I got the same
problem
: Privsep daemon failed to start
[cinder] cinder_store_auth_address = https://dashint.example.com:5000/v3 cinder_store_project_name = service cinder_volume_type = nfstype rootwrap_config = /etc/glance/rootwrap.conf
*cinder_store_user_name = glancecinder_store_password =
glance-password*
cinder_catalog_info = volumev3:cinderv3:internalURL
*###cinder_store_user_name = cinder###cinder_store_password = cinder-password *
I thought once cinder has the capability to read & write into the
backend,
glance will just use it to upload the images!!
Another thing, about the credentials, which ones should I use? Glance's
or
cinder's I thought it should be cinder's !! but in your example @Sean you used {{ glance_keystone_user }} and {{ glance_keystone_password }}
either would work but normally i woudl consider it bad partice for a service to have the username/password for a differnt users account so glance should alwasy use its user to talk to other services when its not using the keystone user token passed in to the api.
Lastly I searched how to launch a privileged container in kolla but I didn't find anything.
i see the issue
https://github.com/openstack/kolla-ansible/blob/stable/yoga/ansible/roles/gl... glace api is only privlaged if you use iscsi for the cinder backend. privileged: "{{ enable_cinder | bool and enable_cinder_backend_iscsi | bool }}" you are using nfs.
you can workaround this in your global.yaml by adding this i think
glance_services: glance-api: privileged: true
alternitivly you can do it via the inventory file by seting that as a host/group var.
Regards.
given it appears to be replacing it directly instead of merging then you need to add the following to global.yaml instead
glance_services: glance-api: container_name: glance_api group: glance-api host_in_groups: "{{ inventory_hostname in glance_api_hosts }}" enabled: true image: "{{ glance_api_image_full }}" environment: "{{ glance_api_container_proxy }}" privileged: true volumes: "{{ glance_api_default_volumes + glance_api_extra_volumes }}" dimensions: "{{ glance_api_dimensions }}" healthcheck: "{{ glance_api_healthcheck }}" haproxy: glance_api: enabled: "{{ enable_glance | bool and not glance_enable_tls_backend | bool }}" mode: "http" external: false port: "{{ glance_api_port }}" frontend_http_extra: - "timeout client {{ haproxy_glance_api_client_timeout }}" backend_http_extra: - "timeout server {{ haproxy_glance_api_server_timeout }}" custom_member_list: "{{ haproxy_members.split(';') }}" glance_api_external: enabled: "{{ enable_glance | bool and not glance_enable_tls_backend | bool }}" mode: "http" external: true port: "{{ glance_api_port }}" frontend_http_extra: - "timeout client {{ haproxy_glance_api_client_timeout }}" backend_http_extra: - "timeout server {{ haproxy_glance_api_server_timeout }}" custom_member_list: "{{ haproxy_members.split(';') }}"
basically copy the default with all teh templates and just update privileged to true
On Thu, 2023-04-13 at 15:23 +0100, wodel youchi wrote:
Hi,
Adding those lines to globals.yml did produce errors, complaining about the enabled line missing
glance_services: glance-api: privileged: true
Then I tried this
glance_services: glance-api: enabled: true privileged: true
Gave these errors fatal: [192.168.2.23]: FAILED! => {"msg": "The conditional check 'item.value.host_in_groups | bool' failed. The error was: error while evaluating conditional (item.value.host_in_groups | bool): 'dict object' has no attribute 'host_in_groups'\n\nThe error appears to be in '/home/deployer/kollavenv/yogavenv/share/kolla-ansible/ansible/roles/glance/tasks/config.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Ensuring config directories exist\n ^ here\n"} fatal: [192.168.2.27]: FAILED! => {"msg": "The conditional check 'item.value.host_in_groups | bool' failed. The error was: error while evaluating conditional (item.value.host_in_groups | bool): 'dict object' has no attribute 'host_in_groups'\n\nThe error appears to be in '/home/deployer/kollavenv/yogavenv/share/kolla-ansible/ansible/roles/glance/tasks/config.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Ensuring config directories exist\n ^ here\n"} fatal: [192.168.2.31]: FAILED! => {"msg": "The conditional check 'item.value.host_in_groups | bool' failed. The error was: error while evaluating conditional (item.value.host_in_groups | bool): 'dict object' has no attribute 'host_in_groups'\n\nThe error appears to be in '/home/deployer/kollavenv/yogavenv/share/kolla-ansible/ansible/roles/glance/tasks/config.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Ensuring config directories exist\n ^ here\n"}
it seems, this configuration is discarding main.yml from https://github.com/openstack/kolla-ansible/blob/stable/yoga/ansible/roles/gl... not just replacing the needed variables.
Regards.
Le jeu. 13 avr. 2023 à 13:00, Sean Mooney smooney@redhat.com a écrit :
On Thu, 2023-04-13 at 11:41 +0100, wodel youchi wrote:
Thanks for the help,
I modified my glance-api.conf like this but no luck, I got the same
problem
: Privsep daemon failed to start
[cinder] cinder_store_auth_address = https://dashint.example.com:5000/v3 cinder_store_project_name = service cinder_volume_type = nfstype rootwrap_config = /etc/glance/rootwrap.conf
*cinder_store_user_name = glancecinder_store_password =
glance-password*
cinder_catalog_info = volumev3:cinderv3:internalURL
*###cinder_store_user_name = cinder###cinder_store_password = cinder-password *
I thought once cinder has the capability to read & write into the
backend,
glance will just use it to upload the images!!
Another thing, about the credentials, which ones should I use? Glance's
or
cinder's I thought it should be cinder's !! but in your example @Sean you used {{ glance_keystone_user }} and {{ glance_keystone_password }}
either would work but normally i woudl consider it bad partice for a service to have the username/password for a differnt users account so glance should alwasy use its user to talk to other services when its not using the keystone user token passed in to the api.
Lastly I searched how to launch a privileged container in kolla but I didn't find anything.
i see the issue
https://github.com/openstack/kolla-ansible/blob/stable/yoga/ansible/roles/gl... glace api is only privlaged if you use iscsi for the cinder backend. privileged: "{{ enable_cinder | bool and enable_cinder_backend_iscsi | bool }}" you are using nfs.
you can workaround this in your global.yaml by adding this i think
glance_services: glance-api: privileged: true
alternitivly you can do it via the inventory file by seting that as a host/group var.
Regards.
On Fri, 2023-04-14 at 12:46 +0100, Sean Mooney wrote:
given it appears to be replacing it directly instead of merging then you need to add the following to global.yaml instead
slight correction you will also need to include the glance-tls-proxy section https://github.com/openstack/kolla-ansible/blob/stable/yoga/ansible/roles/gl... since that is also under glance_services so copy all the highlighted section
glance_services: glance-api: container_name: glance_api group: glance-api host_in_groups: "{{ inventory_hostname in glance_api_hosts }}" enabled: true image: "{{ glance_api_image_full }}" environment: "{{ glance_api_container_proxy }}" privileged: true volumes: "{{ glance_api_default_volumes + glance_api_extra_volumes }}" dimensions: "{{ glance_api_dimensions }}" healthcheck: "{{ glance_api_healthcheck }}" haproxy: glance_api: enabled: "{{ enable_glance | bool and not glance_enable_tls_backend | bool }}" mode: "http" external: false port: "{{ glance_api_port }}" frontend_http_extra: - "timeout client {{ haproxy_glance_api_client_timeout }}" backend_http_extra: - "timeout server {{ haproxy_glance_api_server_timeout }}" custom_member_list: "{{ haproxy_members.split(';') }}" glance_api_external: enabled: "{{ enable_glance | bool and not glance_enable_tls_backend | bool }}" mode: "http" external: true port: "{{ glance_api_port }}" frontend_http_extra: - "timeout client {{ haproxy_glance_api_client_timeout }}" backend_http_extra: - "timeout server {{ haproxy_glance_api_server_timeout }}" custom_member_list: "{{ haproxy_members.split(';') }}"
basically copy the default with all teh templates and just update privileged to true
On Thu, 2023-04-13 at 15:23 +0100, wodel youchi wrote:
Hi,
Adding those lines to globals.yml did produce errors, complaining about the enabled line missing
glance_services: glance-api: privileged: true
Then I tried this
glance_services: glance-api: enabled: true privileged: true
Gave these errors fatal: [192.168.2.23]: FAILED! => {"msg": "The conditional check 'item.value.host_in_groups | bool' failed. The error was: error while evaluating conditional (item.value.host_in_groups | bool): 'dict object' has no attribute 'host_in_groups'\n\nThe error appears to be in '/home/deployer/kollavenv/yogavenv/share/kolla-ansible/ansible/roles/glance/tasks/config.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Ensuring config directories exist\n ^ here\n"} fatal: [192.168.2.27]: FAILED! => {"msg": "The conditional check 'item.value.host_in_groups | bool' failed. The error was: error while evaluating conditional (item.value.host_in_groups | bool): 'dict object' has no attribute 'host_in_groups'\n\nThe error appears to be in '/home/deployer/kollavenv/yogavenv/share/kolla-ansible/ansible/roles/glance/tasks/config.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Ensuring config directories exist\n ^ here\n"} fatal: [192.168.2.31]: FAILED! => {"msg": "The conditional check 'item.value.host_in_groups | bool' failed. The error was: error while evaluating conditional (item.value.host_in_groups | bool): 'dict object' has no attribute 'host_in_groups'\n\nThe error appears to be in '/home/deployer/kollavenv/yogavenv/share/kolla-ansible/ansible/roles/glance/tasks/config.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Ensuring config directories exist\n ^ here\n"}
it seems, this configuration is discarding main.yml from https://github.com/openstack/kolla-ansible/blob/stable/yoga/ansible/roles/gl... not just replacing the needed variables.
Regards.
Le jeu. 13 avr. 2023 à 13:00, Sean Mooney smooney@redhat.com a écrit :
On Thu, 2023-04-13 at 11:41 +0100, wodel youchi wrote:
Thanks for the help,
I modified my glance-api.conf like this but no luck, I got the same
problem
: Privsep daemon failed to start
[cinder] cinder_store_auth_address = https://dashint.example.com:5000/v3 cinder_store_project_name = service cinder_volume_type = nfstype rootwrap_config = /etc/glance/rootwrap.conf
*cinder_store_user_name = glancecinder_store_password =
glance-password*
cinder_catalog_info = volumev3:cinderv3:internalURL
*###cinder_store_user_name = cinder###cinder_store_password = cinder-password *
I thought once cinder has the capability to read & write into the
backend,
glance will just use it to upload the images!!
Another thing, about the credentials, which ones should I use? Glance's
or
cinder's I thought it should be cinder's !! but in your example @Sean you used {{ glance_keystone_user }} and {{ glance_keystone_password }}
either would work but normally i woudl consider it bad partice for a service to have the username/password for a differnt users account so glance should alwasy use its user to talk to other services when its not using the keystone user token passed in to the api.
Lastly I searched how to launch a privileged container in kolla but I didn't find anything.
i see the issue
https://github.com/openstack/kolla-ansible/blob/stable/yoga/ansible/roles/gl... glace api is only privlaged if you use iscsi for the cinder backend. privileged: "{{ enable_cinder | bool and enable_cinder_backend_iscsi | bool }}" you are using nfs.
you can workaround this in your global.yaml by adding this i think
glance_services: glance-api: privileged: true
alternitivly you can do it via the inventory file by seting that as a host/group var.
Regards.
Hi;
I have managed to start the glance-api container in privileged mode, but I still have issues to get glance to use my cinder backend backed by nfs.
When I push an image, now I am getting :
Stderr: '/var/lib/kolla/venv/bin/glance-rootwrap: Unauthorized command: mount -t nfs 20.1.0.32:/kolla_nfs /var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624 f97bf3623b13014f22186b7 (no filter matched)\n'
Any ideas on that?
PS : How did I managed to start the glance-api as privileged container 1 - I edited this file and added this variable vim kollavenv/yogavenv/share/kolla-ansible/ansible/group_vars/all.yml glance_privileged_container: "no"
2 - I edited this file vim kollavenv/yogavenv/share/kolla-ansible/ansible/roles/glance/defaults/main.yml --- glance_services: glance-api: container_name: glance_api group: glance-api host_in_groups: "{{ inventory_hostname in glance_api_hosts }}" enabled: true image: "{{ glance_api_image_full }}" environment: "{{ glance_api_container_proxy }}" *privileged: "{{ enable_cinder | bool and enable_cinder_backend_iscsi | bool or glance_privileged_container | bool }}"* <-------
3 - I added this configuration to my globals.yml vim /etc/yogakolla/globals.yml glance_privileged_container: "yes"
4 - Then I redeployed the glance service kolla-ansible --configdir /etc/yogakolla -i multinode-yoga deploy --tags glance -v
5 - Verifying [root@controllerb ~]# docker inspect glance_api | grep -i pri "IpcMode": "private", *"Pri**vileged": true,* "Propagation": "rprivate" "Propagation": "rprivate" "PS1=$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ ",
Regards.
Le ven. 14 avr. 2023 à 12:48, Sean Mooney smooney@redhat.com a écrit :
On Fri, 2023-04-14 at 12:46 +0100, Sean Mooney wrote:
given it appears to be replacing it directly instead of merging then you
need to add the following to global.yaml instead
slight correction you will also need to include the glance-tls-proxy section
https://github.com/openstack/kolla-ansible/blob/stable/yoga/ansible/roles/gl... since that is also under glance_services so copy all the highlighted section
glance_services: glance-api: container_name: glance_api group: glance-api host_in_groups: "{{ inventory_hostname in glance_api_hosts }}" enabled: true image: "{{ glance_api_image_full }}" environment: "{{ glance_api_container_proxy }}" privileged: true volumes: "{{ glance_api_default_volumes + glance_api_extra_volumes
}}"
dimensions: "{{ glance_api_dimensions }}" healthcheck: "{{ glance_api_healthcheck }}" haproxy: glance_api: enabled: "{{ enable_glance | bool and notglance_enable_tls_backend | bool }}"
mode: "http" external: false port: "{{ glance_api_port }}" frontend_http_extra: - "timeout client {{ haproxy_glance_api_client_timeout }}" backend_http_extra: - "timeout server {{ haproxy_glance_api_server_timeout }}" custom_member_list: "{{ haproxy_members.split(';') }}" glance_api_external: enabled: "{{ enable_glance | bool and notglance_enable_tls_backend | bool }}"
mode: "http" external: true port: "{{ glance_api_port }}" frontend_http_extra: - "timeout client {{ haproxy_glance_api_client_timeout }}" backend_http_extra: - "timeout server {{ haproxy_glance_api_server_timeout }}" custom_member_list: "{{ haproxy_members.split(';') }}"basically copy the default with all teh templates and just update
privileged to true
On Thu, 2023-04-13 at 15:23 +0100, wodel youchi wrote:
Hi,
Adding those lines to globals.yml did produce errors, complaining
about the
enabled line missing
glance_services: glance-api: privileged: true
Then I tried this
glance_services: glance-api: enabled: true privileged: true
Gave these errors fatal: [192.168.2.23]: FAILED! => {"msg": "The conditional check 'item.value.host_in_groups | bool' failed. The error was: error while evaluating conditional (item.value.host_in_groups | bool): 'dict
object'
has no attribute 'host_in_groups'\n\nThe error appears to be in
'/home/deployer/kollavenv/yogavenv/share/kolla-ansible/ansible/roles/glance/tasks/config.yml':
line 2, column 3, but may\nbe elsewhere in the file depending on the
exact
syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Ensuring config directories exist\n ^ here\n"} fatal: [192.168.2.27]: FAILED! => {"msg": "The conditional check 'item.value.host_in_groups | bool' failed. The error was: error while evaluating conditional (item.value.host_in_groups | bool): 'dict
object'
has no attribute 'host_in_groups'\n\nThe error appears to be in
'/home/deployer/kollavenv/yogavenv/share/kolla-ansible/ansible/roles/glance/tasks/config.yml':
line 2, column 3, but may\nbe elsewhere in the file depending on the
exact
syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Ensuring config directories exist\n ^ here\n"} fatal: [192.168.2.31]: FAILED! => {"msg": "The conditional check 'item.value.host_in_groups | bool' failed. The error was: error while evaluating conditional (item.value.host_in_groups | bool): 'dict
object'
has no attribute 'host_in_groups'\n\nThe error appears to be in
'/home/deployer/kollavenv/yogavenv/share/kolla-ansible/ansible/roles/glance/tasks/config.yml':
line 2, column 3, but may\nbe elsewhere in the file depending on the
exact
syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Ensuring config directories exist\n ^ here\n"}
it seems, this configuration is discarding main.yml from
https://github.com/openstack/kolla-ansible/blob/stable/yoga/ansible/roles/gl...
not just replacing the needed variables.
Regards.
Le jeu. 13 avr. 2023 à 13:00, Sean Mooney smooney@redhat.com a
écrit :
On Thu, 2023-04-13 at 11:41 +0100, wodel youchi wrote:
Thanks for the help,
I modified my glance-api.conf like this but no luck, I got the same
problem
: Privsep daemon failed to start
[cinder] cinder_store_auth_address = https://dashint.example.com:5000/v3 cinder_store_project_name = service cinder_volume_type = nfstype rootwrap_config = /etc/glance/rootwrap.conf
*cinder_store_user_name = glancecinder_store_password =
glance-password*
cinder_catalog_info = volumev3:cinderv3:internalURL
*###cinder_store_user_name = cinder###cinder_store_password = cinder-password *
I thought once cinder has the capability to read & write into the
backend,
glance will just use it to upload the images!!
Another thing, about the credentials, which ones should I use?
Glance's
or
cinder's I thought it should be cinder's !! but in your example
@Sean you
used {{ glance_keystone_user }} and {{ glance_keystone_password }}
either would work but normally i woudl consider it bad partice for a service to have the username/password for a differnt users account so glance should alwasy use its user to talk to other services when
its
not using the keystone user token passed in to the api.
Lastly I searched how to launch a privileged container in kolla
but I
didn't find anything.
i see the issue
https://github.com/openstack/kolla-ansible/blob/stable/yoga/ansible/roles/gl...
glace api is only privlaged if you use iscsi for the cinder backend. privileged: "{{ enable_cinder | bool and enable_cinder_backend_iscsi
|
bool }}" you are using nfs.
you can workaround this in your global.yaml by adding this i think
glance_services: glance-api: privileged: true
alternitivly you can do it via the inventory file by seting that as a host/group var.
Regards.
Hi,
I managed to get another step forward, but I hit another wall. Glance-api was trying to mount the NFS share on the controller node, not on the compute, so I installed nfs-utils on it, now I have this error :
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder
[req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Exception while accessing to cinder volume 3d4712ef-7cb5-4a0b-bc1e-cfbebd8fa902.: FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder Traceback (most recent call last): 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 788, in _open_cinder_volume 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder device = connect_volume_nfs() 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 391, in inner 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder return f(*args, **kwargs) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 786, in connect_volume_nfs 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder root_helper, options) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 359, in mount 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder rootwrap_helper, options) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 247, in mount 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder os.makedirs(mountpoint) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/usr/lib64/python3.6/os.py", line 220, in makedirs 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder mkdir(name, mode) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder 2023-04-17 16:33:08.906 62 ERROR glance_store._drivers.cinder [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Failed to write to volume 3d4712ef-7cb5-4a0b-bc1e-cfbebd8fa902.: FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.939 62 ERROR glance.api.v2.image_data [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Failed to upload image data due to internal error: FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Caught error: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7': FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi Traceback (most recent call last): 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1332, in __call__ 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi request, **action_args) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1370, in dispatch 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return method(*args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/utils.py", line 414, in wrapped 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 303, in upload 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self._restore(image_repo, image) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self.force_reraise() 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise self.value 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 163, in upload 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 497, in set_data 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self.force_reraise() 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise self.value 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 444, in set_data 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi set_active=set_active) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/quota/__init__.py", line 323, in set_data 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi set_active=set_active) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 585, in set_data 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 485, in _upload_to_store 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi verifier=verifier) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 399, in add_with_multihash 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 481, in store_add_to_backend_with_multihash 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/driver.py", line 279, in add_adapter 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/capabilities.py", line 176, in op_checker 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 1028, in add 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise errors.get(e.errno, e) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 985, in add 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi with self._open_cinder_volume(client, volume, 'wb') as f: 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return next(self.gen) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 788, in _open_cinder_volume 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi device = connect_volume_nfs() 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 391, in inner 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return f(*args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 786, in connect_volume_nfs 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi root_helper, options) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 359, in mount 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi rootwrap_helper, options) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 247, in mount 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi os.makedirs(mountpoint) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/usr/lib64/python3.6/os.py", line 220, in makedirs 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi mkdir(name, mode) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi
Regards.
https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail Virus-free.www.avast.com https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
On Mon, 2023-04-17 at 16:43 +0100, wodel youchi wrote:
Hi,
I managed to get another step forward, but I hit another wall. Glance-api was trying to mount the NFS share on the controller node
corret galnce-api si not runnign on a compute at least not by default. and galnce cannot assume that its runnign on the same hosta s cinder so it cannot assume any mounting done by cinder can be reused.
, not on the compute, so I installed nfs-utils on it, now I have this error :
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder
[req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Exception while accessing to cinder volume 3d4712ef-7cb5-4a0b-bc1e-cfbebd8fa902.: FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder Traceback (most recent call last): 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 788, in _open_cinder_volume 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder device = connect_volume_nfs() 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 391, in inner 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder return f(*args, **kwargs) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 786, in connect_volume_nfs 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder root_helper, options) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 359, in mount 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder rootwrap_helper, options) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 247, in mount 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder os.makedirs(mountpoint) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/usr/lib64/python3.6/os.py", line 220, in makedirs 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder mkdir(name, mode) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder 2023-04-17 16:33:08.906 62 ERROR glance_store._drivers.cinder [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Failed to write to volume 3d4712ef-7cb5-4a0b-bc1e-cfbebd8fa902.: FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.939 62 ERROR glance.api.v2.image_data [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Failed to upload image data due to internal error: FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Caught error: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7': FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi Traceback (most recent call last): 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1332, in __call__ 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi request, **action_args) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1370, in dispatch 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return method(*args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/utils.py", line 414, in wrapped 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 303, in upload 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self._restore(image_repo, image) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self.force_reraise() 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise self.value 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 163, in upload 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 497, in set_data 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self.force_reraise() 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise self.value 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 444, in set_data 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi set_active=set_active) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/quota/__init__.py", line 323, in set_data 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi set_active=set_active) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 585, in set_data 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 485, in _upload_to_store 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi verifier=verifier) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 399, in add_with_multihash 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 481, in store_add_to_backend_with_multihash 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/driver.py", line 279, in add_adapter 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/capabilities.py", line 176, in op_checker 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 1028, in add 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise errors.get(e.errno, e) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 985, in add 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi with self._open_cinder_volume(client, volume, 'wb') as f: 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return next(self.gen) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 788, in _open_cinder_volume 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi device = connect_volume_nfs() 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 391, in inner 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return f(*args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 786, in connect_volume_nfs 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi root_helper, options) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 359, in mount 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi rootwrap_helper, options) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 247, in mount 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi os.makedirs(mountpoint) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/usr/lib64/python3.6/os.py", line 220, in makedirs 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi mkdir(name, mode) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi
Regards.
https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail Virus-free.www.avast.com https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Hi,
On Mon, Apr 17, 2023 at 9:19 PM wodel youchi wodel.youchi@gmail.com wrote:
Hi,
I managed to get another step forward, but I hit another wall. Glance-api was trying to mount the NFS share on the controller node, not on the compute, so I installed nfs-utils on it, now I have this error :
Glance will mount the share on the node in which it's running. The default location for mounting is "/var/lib/glance/mnt" and can be configured via config option "cinder_mount_point_base"[1] Since Glance is running on the controller node in your case, it will mount the share on the controller.
2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder
[req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Exception while accessing to cinder volume 3d4712ef-7cb5-4a0b-bc1e-cfbebd8fa902.: FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder Traceback (most recent call last): 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 788, in _open_cinder_volume 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder device = connect_volume_nfs() 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 391, in inner 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder return f(*args, **kwargs) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 786, in connect_volume_nfs 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder root_helper, options) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 359, in mount 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder rootwrap_helper, options) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 247, in mount 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder os.makedirs(mountpoint) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder File "/usr/lib64/python3.6/os.py", line 220, in makedirs 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder mkdir(name, mode) 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.814 62 ERROR glance_store._drivers.cinder 2023-04-17 16:33:08.906 62 ERROR glance_store._drivers.cinder [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Failed to write to volume 3d4712ef-7cb5-4a0b-bc1e-cfbebd8fa902.: FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.939 62 ERROR glance.api.v2.image_data [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Failed to upload image data due to internal error: FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi [req-35f49fc4-15bc-4d3b-b93e-e039c1d0f3fa 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Caught error: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7': FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi Traceback (most recent call last): 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1332, in __call__ 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi request, **action_args) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1370, in dispatch 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return method(*args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/utils.py", line 414, in wrapped 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 303, in upload 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self._restore(image_repo, image) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self.force_reraise() 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise self.value 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 163, in upload 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 497, in set_data 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self.force_reraise() 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise self.value 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 444, in set_data 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi set_active=set_active) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/quota/__init__.py", line 323, in set_data 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi set_active=set_active) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 585, in set_data 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 485, in _upload_to_store 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi verifier=verifier) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 399, in add_with_multihash 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 481, in store_add_to_backend_with_multihash 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/driver.py", line 279, in add_adapter 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/capabilities.py", line 176, in op_checker 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 1028, in add 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi raise errors.get(e.errno, e) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 985, in add 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi with self._open_cinder_volume(client, volume, 'wb') as f: 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return next(self.gen) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 788, in _open_cinder_volume 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi device = connect_volume_nfs() 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 391, in inner 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi return f(*args, **kwargs) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 786, in connect_volume_nfs 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi root_helper, options) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 359, in mount 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi rootwrap_helper, options) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/common/fs_mount.py", line 247, in mount 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi os.makedirs(mountpoint) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi File "/usr/lib64/python3.6/os.py", line 220, in makedirs 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi mkdir(name, mode) 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi FileExistsError: [Errno 17] File exists: '/var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624f97bf3623b13014f22186b7' 2023-04-17 16:33:08.957 62 ERROR glance.common.wsgi
This is a strange error, we acquire a lock on the export path before it is mounted[2] and again acquire a threading lock on the mountpoint[3]. Then we check if the path is already mounted and only create the directory. we also remove the directory during unmount[4]. Are you using multiple glance processes with concurrent operations running? I can see we have a limitation because of the threading lock that only prevents access to the thread of the same process.
[1] https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc1... [2] https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc1... [3] https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc1... [4] https://github.com/openstack/glance_store/blob/9bd9cf4fcd8a0aedc98fafb983fc1...
Thanks Rajat Dhasmana
Regards.
https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail Virus-free.www.avast.com https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#m_-7349147206883296287_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
participants (4)
-
Alan Bishop
-
Rajat Dhasmana
-
Sean Mooney
-
wodel youchi