On Wed, Apr 12, 2023 at 10:41 AM wodel youchi wodel.youchi@gmail.com wrote:
Hi,
I am trying to configure glance to use cinder as a backend.
This is my glance-api.conf [cinder] cinder_store_auth_address = https://dashint.example.com:5000/v3 cinder_store_user_name = cinder cinder_store_password = cinderpass cinder_store_project_name = service cinder_volume_type = nfstype rootwrap_config = /etc/glance/rootwrap.conf
==> /var/log/kolla/glance/glance-api.log <==
2023-04-12 18:02:20.842 64 INFO oslo.privsep.daemon [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Spawned new privsep daemon via rootwrap 2023-04-12 18:02:20.733 360 INFO oslo.privsep.daemon [-] privsep daemon starting 2023-04-12 18:02:20.735 360 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
*2023-04-12 18:02:20.737 360 ERROR oslo.privsep.daemon [-] [Errno 1] Operation not permitted Traceback (most recent call last): * File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 584, in helper_main Daemon(channel, context).run() File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 394, in run self._drop_privs() File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 430, in _drop_privs capabilities.drop_all_caps_except(self.caps, self.caps, []) File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/capabilities.py", line 156, in drop_all_caps_except raise OSError(errno, os.strerror(errno)) PermissionError: [Errno 1] Operation not permitted 2023-04-12 18:02:20.844 64 WARNING oslo_privsep.comm [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Unexpected error: <class 'BrokenPipeError'>: BrokenPipeError: [Errno 32] Broken pipe 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Error while sending initial PING to privsep: [Errno 32] Broken pipe: BrokenPipeError: [Errno 32] Broken pipe 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon Traceback (most recent call last): 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 192, in exchange_ping 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon reply = self.send_recv((comm.Message.PING.value,)) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/comm.py", line 186, in send_recv 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon self.writer.send((myid, msg)) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/comm.py", line 60, in send 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon self.writesock.sendall(buf) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py", line 407, in sendall 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon tail = self.send(data, flags) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py", line 401, in send 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon return self._send_loop(self.fd.send, data, flags) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py", line 388, in _send_loop 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon return send_method(data, *args) 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon BrokenPipeError: [Errno 32] Broken pipe 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon 2023-04-12 18:02:20.846 64 CRITICAL oslo.privsep.daemon [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - defau lt default] Privsep daemon failed to start 2023-04-12 18:02:20.847 64 ERROR glance_store._drivers.cinder [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Failed to write to volume 46316c12-6c24-40af-afde-1c16edd616b6.: oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start 2023-04-12 18:02:20.890 64 ERROR glance.api.v2.image_data [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - def ault default] *Failed to upload image data due to internal error: oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start * 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default d efault] Caught error: Privsep daemon failed to start: oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi Traceback (most recent call last): 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1332, in __call__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi request, **action_args) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", line 1370, in dispatch 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return method(*args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/utils.py", line 414, in wrapped 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 303, in upload 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self._restore(image_repo, image) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.force_reraise() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise self.value 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 163, in upload 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 497, in set_data 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.force_reraise() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise self.value 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line 444, in set_data 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi set_active=set_active) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/quota/__init__.py", line 323, in set_data 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi set_active=set_active) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 585, in set_data 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line 485, in _upload_to_store 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi verifier=verifier) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 399, in add_with_multihash 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", line 481, in store_add_to_backe nd_with_multihash 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/driver.py", line 279, in add_adapter 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/capabilities.py", line 176, in op_checker 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 985, in add 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi with self._open_cinder_volume(client, volume, 'wb') as f: 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return next(self.gen) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 739, in _open_cinder_vol ume 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi root_helper, my_ip, use_multipath, enforce_multipath, host=host) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/utils.py", line 169, in trace_logging_wrapper 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return f(*args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connector.py", line 240, in get_connector_pr operties 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi execute=execute)) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 70, in get_connec tor_properties 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi initiator = iscsi.get_initiator() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 963, in get_initi ator 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi root_helper=self._root_helper) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/executor.py", line 53, in _execute 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi result = self.__execute(*args, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 172, in execute 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return execute_root(*cmd, **kwargs) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 269, in _wrap 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.start() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 283, in start 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi channel = daemon.RootwrapClientChannel(context=self) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 374, in __init__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi super(RootwrapClientChannel, self).__init__(sock, context) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 187, in __init__ 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.exchange_ping() 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 201, in exchange_ping 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise FailedToDropPrivileges(msg) 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi *oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start * 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi 2023-04-12 18:02:20.927 64 INFO eventlet.wsgi.server [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] 20.3.0.34,127.0.0.1 - - [12/Apr/2023 18:02:20] "PUT /v2/images/52ed7ed7-330e-4249-abb9-5ec99712846f/file HTTP/1.1" 500 430 2.727683
It seems like a lack of privileges, any ideas?
Yes, the glance-api container itself must run with "privileged: true" when glance is using cinder for a backend. For reference, you can see how TripleO handles this:
https://github.com/openstack/tripleo-heat-templates/blob/2e6d826debd6099b3d8...
Alan
Regards.