[kolla-ansible][yoga] Glance backend cinder Privsep daemon failed to start operation not permitted
Alan Bishop
abishop at redhat.com
Wed Apr 12 18:12:24 UTC 2023
On Wed, Apr 12, 2023 at 10:41 AM wodel youchi <wodel.youchi at gmail.com>
wrote:
> Hi,
>
> I am trying to configure glance to use cinder as a backend.
>
> This is my glance-api.conf
> [cinder]
> cinder_store_auth_address = https://dashint.example.com:5000/v3
> cinder_store_user_name = cinder
> cinder_store_password = cinderpass
> cinder_store_project_name = service
> cinder_volume_type = nfstype
> rootwrap_config = /etc/glance/rootwrap.conf
>
>
>
> ==> /var/log/kolla/glance/glance-api.log <==
>> 2023-04-12 18:02:20.842 64 INFO oslo.privsep.daemon
>> [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 - default default] Spawned new privsep
>> daemon via rootwrap
>> 2023-04-12 18:02:20.733 360 INFO oslo.privsep.daemon [-] privsep daemon
>> starting
>> 2023-04-12 18:02:20.735 360 INFO oslo.privsep.daemon [-] privsep process
>> running with uid/gid: 0/0
>>
>> *2023-04-12 18:02:20.737 360 ERROR oslo.privsep.daemon [-] [Errno 1]
>> Operation not permitted Traceback (most recent call last): *
>> File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py",
>> line 584, in helper_main
>> Daemon(channel, context).run()
>> File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py",
>> line 394, in run
>> self._drop_privs()
>> File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py",
>> line 430, in _drop_privs
>> capabilities.drop_all_caps_except(self.caps, self.caps, [])
>> File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/capabilities.py",
>> line 156, in drop_all_caps_except
>> raise OSError(errno, os.strerror(errno))
>> PermissionError: [Errno 1] Operation not permitted
>> 2023-04-12 18:02:20.844 64 WARNING oslo_privsep.comm
>> [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 - default
>> default] Unexpected error: <class 'BrokenPipeError'>: BrokenPipeError:
>> [Errno 32] Broken pipe
>> 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon
>> [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 - default
>> default] Error while sending initial PING to privsep: [Errno 32] Broken
>> pipe: BrokenPipeError: [Errno 32] Broken pipe
>> 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon Traceback (most
>> recent call last):
>> 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py",
>> line 192, in exchange_ping
>> 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon reply =
>> self.send_recv((comm.Message.PING.value,))
>> 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/comm.py",
>> line 186, in send_recv
>> 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon
>> self.writer.send((myid, msg))
>> 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/comm.py",
>> line 60, in send
>> 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon
>> self.writesock.sendall(buf)
>> 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py",
>> line 407, in sendall
>> 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon tail =
>> self.send(data, flags)
>> 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py",
>> line 401, in send
>> 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon return
>> self._send_loop(self.fd.send, data, flags)
>> 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py",
>> line 388, in _send_loop
>> 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon return
>> send_method(data, *args)
>> 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon BrokenPipeError:
>> [Errno 32] Broken pipe
>> 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon
>> 2023-04-12 18:02:20.846 64 CRITICAL oslo.privsep.daemon
>> [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 - defau
>> lt default] Privsep daemon failed to start
>> 2023-04-12 18:02:20.847 64 ERROR glance_store._drivers.cinder
>> [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 -
>> default default] Failed to write to volume
>> 46316c12-6c24-40af-afde-1c16edd616b6.:
>> oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start
>> 2023-04-12 18:02:20.890 64 ERROR glance.api.v2.image_data
>> [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 - def
>> ault default] *Failed to upload image data due to internal error:
>> oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start *
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
>> [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 - default d
>> efault] Caught error: Privsep daemon failed to start:
>> oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi Traceback (most
>> recent call last):
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py",
>> line 1332, in __call__
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi request,
>> **action_args)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py",
>> line 1370, in dispatch
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return
>> method(*args, **kwargs)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/utils.py",
>> line 414, in wrapped
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return func(self,
>> req, *args, **kwargs)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py",
>> line 303, in upload
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
>> self._restore(image_repo, image)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 227, in __exit__
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
>> self.force_reraise()
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 200, in force_reraise
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise self.value
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py",
>> line 163, in upload
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
>> image.set_data(data, size, backend=backend)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line
>> 497, in set_data
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
>> _send_notification(notify_error, 'image.upload', msg)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 227, in __exit__
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
>> self.force_reraise()
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 200, in force_reraise
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise self.value
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line
>> 444, in set_data
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
>> set_active=set_active)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/quota/__init__.py",
>> line 323, in set_data
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
>> set_active=set_active)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line
>> 585, in set_data
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
>> self._upload_to_store(data, verifier, backend, size)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line
>> 485, in _upload_to_store
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
>> verifier=verifier)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py",
>> line 399, in add_with_multihash
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi image_id, data,
>> size, hashing_algo, store, context, verifier)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py",
>> line 481, in store_add_to_backe
>> nd_with_multihash
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi image_id, data,
>> size, hashing_algo, context=context, verifier=verifier)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/driver.py",
>> line 279, in add_adapter
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi metadata_dict) =
>> store_add_fun(*args, **kwargs)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/capabilities.py",
>> line 176, in op_checker
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return
>> store_op_fun(store, *args, **kwargs)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py",
>> line 985, in add
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi with
>> self._open_cinder_volume(client, volume, 'wb') as f:
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return
>> next(self.gen)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py",
>> line 739, in _open_cinder_vol
>> ume
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi root_helper,
>> my_ip, use_multipath, enforce_multipath, host=host)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/utils.py", line
>> 169, in trace_logging_wrapper
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return f(*args,
>> **kwargs)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connector.py",
>> line 240, in get_connector_pr
>> operties
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi execute=execute))
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py",
>> line 70, in get_connec
>> tor_properties
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi initiator =
>> iscsi.get_initiator()
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py",
>> line 963, in get_initi
>> ator
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
>> root_helper=self._root_helper)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/executor.py",
>> line 53, in _execute
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi result =
>> self.__execute(*args, **kwargs)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py",
>> line 172, in execute
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return
>> execute_root(*cmd, **kwargs)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py",
>> line 269, in _wrap
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.start()
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py",
>> line 283, in start
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi channel =
>> daemon.RootwrapClientChannel(context=self)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py",
>> line 374, in __init__
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
>> super(RootwrapClientChannel, self).__init__(sock, context)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py",
>> line 187, in __init__
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
>> self.exchange_ping()
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py",
>> line 201, in exchange_ping
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise
>> FailedToDropPrivileges(msg)
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi *oslo_privsep.daemon.FailedToDropPrivileges:
>> Privsep daemon failed to start *
>> 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
>> 2023-04-12 18:02:20.927 64 INFO eventlet.wsgi.server
>> [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 - default
>> default] 20.3.0.34,127.0.0.1 - - [12/Apr/2023 18:02:20] "PUT
>> /v2/images/52ed7ed7-330e-4249-abb9-5ec99712846f/file HTTP/1.1" 500 430
>> 2.727683
>>
>
> It seems like a lack of privileges, any ideas?
>
Yes, the glance-api container itself must run with "privileged: true" when
glance is using cinder for a backend. For reference, you can see how
TripleO handles this:
https://github.com/openstack/tripleo-heat-templates/blob/2e6d826debd6099b3d85d0268430541b01560139/deployment/glance/glance-api-container-puppet.yaml#L790
Alan
>
> Regards.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230412/127c8e4d/attachment-0001.htm>
More information about the openstack-discuss
mailing list