[kolla-ansible][yoga] Glance backend cinder Privsep daemon failed to start operation not permitted
Sean Mooney
smooney at redhat.com
Wed Apr 12 19:13:03 UTC 2023
On Wed, 2023-04-12 at 11:12 -0700, Alan Bishop wrote:
> On Wed, Apr 12, 2023 at 10:41 AM wodel youchi <wodel.youchi at gmail.com>
> wrote:
>
> > Hi,
> >
> > I am trying to configure glance to use cinder as a backend.
> >
> > This is my glance-api.conf
> > [cinder]
> > cinder_store_auth_address = https://dashint.example.com:5000/v3
> > cinder_store_user_name = cinder
> > cinder_store_password = cinderpass
> > cinder_store_project_name = service
> > cinder_volume_type = nfstype
> > rootwrap_config = /etc/glance/rootwrap.conf
> >
> >
> >
> > ==> /var/log/kolla/glance/glance-api.log <==
> > > 2023-04-12 18:02:20.842 64 INFO oslo.privsep.daemon
> > > [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4
> > > b0f76b5c6dcb457fa716762bbf954837 - default default] Spawned new privsep
> > > daemon via rootwrap
> > > 2023-04-12 18:02:20.733 360 INFO oslo.privsep.daemon [-] privsep daemon
> > > starting
> > > 2023-04-12 18:02:20.735 360 INFO oslo.privsep.daemon [-] privsep process
> > > running with uid/gid: 0/0
> > >
> > > *2023-04-12 18:02:20.737 360 ERROR oslo.privsep.daemon [-] [Errno 1]
> > > Operation not permitted Traceback (most recent call last): *
> > > File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py",
> > > line 584, in helper_main
> > > Daemon(channel, context).run()
> > > File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py",
> > > line 394, in run
> > > self._drop_privs()
> > > File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py",
> > > line 430, in _drop_privs
> > > capabilities.drop_all_caps_except(self.caps, self.caps, [])
> > > File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/capabilities.py",
> > > line 156, in drop_all_caps_except
> > > raise OSError(errno, os.strerror(errno))
> > > PermissionError: [Errno 1] Operation not permitted
> > > 2023-04-12 18:02:20.844 64 WARNING oslo_privsep.comm
> > > [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4
> > > b0f76b5c6dcb457fa716762bbf954837 - default
> > > default] Unexpected error: <class 'BrokenPipeError'>: BrokenPipeError:
> > > [Errno 32] Broken pipe
> > > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon
> > > [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4
> > > b0f76b5c6dcb457fa716762bbf954837 - default
> > > default] Error while sending initial PING to privsep: [Errno 32] Broken
> > > pipe: BrokenPipeError: [Errno 32] Broken pipe
> > > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon Traceback (most
> > > recent call last):
> > > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py",
> > > line 192, in exchange_ping
> > > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon reply =
> > > self.send_recv((comm.Message.PING.value,))
> > > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/comm.py",
> > > line 186, in send_recv
> > > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon
> > > self.writer.send((myid, msg))
> > > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/comm.py",
> > > line 60, in send
> > > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon
> > > self.writesock.sendall(buf)
> > > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py",
> > > line 407, in sendall
> > > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon tail =
> > > self.send(data, flags)
> > > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py",
> > > line 401, in send
> > > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon return
> > > self._send_loop(self.fd.send, data, flags)
> > > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py",
> > > line 388, in _send_loop
> > > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon return
> > > send_method(data, *args)
> > > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon BrokenPipeError:
> > > [Errno 32] Broken pipe
> > > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon
> > > 2023-04-12 18:02:20.846 64 CRITICAL oslo.privsep.daemon
> > > [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4
> > > b0f76b5c6dcb457fa716762bbf954837 - defau
> > > lt default] Privsep daemon failed to start
> > > 2023-04-12 18:02:20.847 64 ERROR glance_store._drivers.cinder
> > > [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4
> > > b0f76b5c6dcb457fa716762bbf954837 -
> > > default default] Failed to write to volume
> > > 46316c12-6c24-40af-afde-1c16edd616b6.:
> > > oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start
> > > 2023-04-12 18:02:20.890 64 ERROR glance.api.v2.image_data
> > > [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4
> > > b0f76b5c6dcb457fa716762bbf954837 - def
> > > ault default] *Failed to upload image data due to internal error:
> > > oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start *
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
> > > [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4
> > > b0f76b5c6dcb457fa716762bbf954837 - default d
> > > efault] Caught error: Privsep daemon failed to start:
> > > oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi Traceback (most
> > > recent call last):
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py",
> > > line 1332, in __call__
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi request,
> > > **action_args)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py",
> > > line 1370, in dispatch
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return
> > > method(*args, **kwargs)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/utils.py",
> > > line 414, in wrapped
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return func(self,
> > > req, *args, **kwargs)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py",
> > > line 303, in upload
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
> > > self._restore(image_repo, image)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> > > line 227, in __exit__
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
> > > self.force_reraise()
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> > > line 200, in force_reraise
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise self.value
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py",
> > > line 163, in upload
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
> > > image.set_data(data, size, backend=backend)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line
> > > 497, in set_data
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
> > > _send_notification(notify_error, 'image.upload', msg)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> > > line 227, in __exit__
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
> > > self.force_reraise()
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> > > line 200, in force_reraise
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise self.value
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line
> > > 444, in set_data
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
> > > set_active=set_active)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/quota/__init__.py",
> > > line 323, in set_data
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
> > > set_active=set_active)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line
> > > 585, in set_data
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
> > > self._upload_to_store(data, verifier, backend, size)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line
> > > 485, in _upload_to_store
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
> > > verifier=verifier)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py",
> > > line 399, in add_with_multihash
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi image_id, data,
> > > size, hashing_algo, store, context, verifier)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py",
> > > line 481, in store_add_to_backe
> > > nd_with_multihash
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi image_id, data,
> > > size, hashing_algo, context=context, verifier=verifier)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/driver.py",
> > > line 279, in add_adapter
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi metadata_dict) =
> > > store_add_fun(*args, **kwargs)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/capabilities.py",
> > > line 176, in op_checker
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return
> > > store_op_fun(store, *args, **kwargs)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py",
> > > line 985, in add
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi with
> > > self._open_cinder_volume(client, volume, 'wb') as f:
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return
> > > next(self.gen)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py",
> > > line 739, in _open_cinder_vol
> > > ume
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi root_helper,
> > > my_ip, use_multipath, enforce_multipath, host=host)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/utils.py", line
> > > 169, in trace_logging_wrapper
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return f(*args,
> > > **kwargs)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connector.py",
> > > line 240, in get_connector_pr
> > > operties
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi execute=execute))
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py",
> > > line 70, in get_connec
> > > tor_properties
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi initiator =
> > > iscsi.get_initiator()
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py",
> > > line 963, in get_initi
> > > ator
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
> > > root_helper=self._root_helper)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/executor.py",
> > > line 53, in _execute
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi result =
> > > self.__execute(*args, **kwargs)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py",
> > > line 172, in execute
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return
> > > execute_root(*cmd, **kwargs)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py",
> > > line 269, in _wrap
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.start()
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py",
> > > line 283, in start
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi channel =
> > > daemon.RootwrapClientChannel(context=self)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py",
> > > line 374, in __init__
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
> > > super(RootwrapClientChannel, self).__init__(sock, context)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py",
> > > line 187, in __init__
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
> > > self.exchange_ping()
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File
> > > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py",
> > > line 201, in exchange_ping
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise
> > > FailedToDropPrivileges(msg)
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi *oslo_privsep.daemon.FailedToDropPrivileges:
> > > Privsep daemon failed to start *
> > > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi
> > > 2023-04-12 18:02:20.927 64 INFO eventlet.wsgi.server
> > > [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4
> > > b0f76b5c6dcb457fa716762bbf954837 - default
> > > default] 20.3.0.34,127.0.0.1 - - [12/Apr/2023 18:02:20] "PUT
> > > /v2/images/52ed7ed7-330e-4249-abb9-5ec99712846f/file HTTP/1.1" 500 430
> > > 2.727683
> > >
> >
> > It seems like a lack of privileges, any ideas?
> >
>
> Yes, the glance-api container itself must run with "privileged: true" when
> glance is using cinder for a backend. For reference, you can see how
> TripleO handles this:
>
> https://github.com/openstack/tripleo-heat-templates/blob/2e6d826debd6099b3d85d0268430541b01560139/deployment/glance/glance-api-container-puppet.yaml#L790
that should work in yoga i have my home cloud using glance-api with cinder back storage
sean at cloud:~/repos/kolla-ansible$ cat /etc/kolla/config/glance/glance-api.conf
[DEFAULT]
enabled_import_methods = ['glance-direct','web-download','copy-image']
enabled_backends = local:cinder
cinder_catalog_info = volumev3:cinderv3:internalURL
show_multiple_locations = True
show_image_direct_url = True
[glance_store]
default_backend = local
default_store = cinder
stores = cinder
[keystone_authtoken]
interface = internal
[local]
cinder_volume_type = local_storage
description = LVM based cinder store
cinder_catalog_info = volumev3:cinderv3:internalURL
cinder_store_auth_address = {{ keystone_internal_url }}
cinder_store_project_name = service
cinder_store_user_name = {{ glance_keystone_user }}
cinder_store_password = {{ glance_keystone_password }}
my glance options in gobal.yamlare
########################
# Glance - Image Options
########################
# Configure image backend.
#glance_backend_ceph: "no"
glance_backend_file: "no"
#glance_backend_swift: "no"
#glance_backend_vmware: "no"
#enable_glance_image_cache: "no"
glance_enable_property_protection: "no"
glance_enable_interoperable_image_import: "yes"
# Configure glance upgrade option.
# Due to this feature being experimental in glance,
# the default value is "no".
#glance_enable_rolling_upgrade: "no"
and cinder lvm
# Cinder - Block Storage Options
################################
# Enable / disable Cinder backends
#cinder_backend_ceph: "no"
#cinder_backend_vmwarevc_vmdk: "no"
#cinder_backend_vmware_vstorage_object: "no"
cinder_volume_group: "cinder-volumes"
# Valid options are [ '', redis, etcd ]
cinder_coordination_backend: ""
on the cidner side i have
sean at cloud:~/repos/kolla-ansible$ cat /etc/kolla/config/cinder.conf
[DEFAULT]
allowed_direct_url_schemes = cinder
[lvm-1]
image_upload_use_cinder_backend = True
image_upload_use_internal_tenant = True
otherwise i requried no other change in kolla-ansible stable/yoga
commit bf1e29c87b52cd4a75f484c1f16c265fdb09b5fa (HEAD -> stable/yoga, origin/stable/yoga)
Merge: 4ee8667f1 f4e6dcc3b
Author: Zuul <zuul at review.opendev.org>
Date: Tue Sep 20 10:00:54 2022 +0000
Merge "Fix prometheus-msteams image name" into stable/yoga
perhaps the nfs type has extra requiement btu lvm should work.
>
> Alan
>
>
>
> >
> > Regards.
> >
> >
More information about the openstack-discuss
mailing list