[kolla[ Issue running masakari-hostmonitor
Hello, I am running Kolla-12.0.0 with Openstack Victoria. I have mostly successfully deployed masakari. The instance monitoring works, I can set up segments in Horizon and via CLI. But I can't get masakari-hostmonitor to start: 2021-12-02 16:50:44.999 7 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'privsep-helper', '--config-file', '/etc/masakari-monitors/masakari -monitors.conf', '--privsep_context', 'masakarimonitors.privsep.monitors_priv', '--privsep_sock_path', '/tmp/tmpf_65hc7m/privsep.sock'] 2021-12-02 16:50:44.892 261 INFO oslo.privsep.daemon [-] privsep daemon starting 2021-12-02 16:50:44.898 261 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2021-12-02 16:50:44.902 261 ERROR oslo.privsep.daemon [-] [Errno 1] Operation not permitted Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 593, in helper_main Daemon(channel, context).run() File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 403, in run self._drop_privs() File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 439, in _drop_privs capabilities.drop_all_caps_except(self.caps, self.caps, []) File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/capabilities.py", line 156, in drop_all_caps_except raise OSError(errno, os.strerror(errno)) PermissionError: [Errno 1] Operation not permitted 2021-12-02 16:50:45.001 7 DEBUG oslo_privsep.comm [-] EOF on privsep read channel _reader_main /var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep /comm.py:149 2021-12-02 16:50:45.475 7 WARNING oslo.privsep.daemon [-] privsep log: [Errno 1] Operation not permitted 2021-12-02 16:50:45.565 7 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap 2021-12-02 16:50:45.565 7 DEBUG oslo.privsep.daemon [-] Accepted privsep connection to /tmp/tmpf_65hc7m/privsep.sock __init__ /var/lib/kolla/venv/lib/pytho n3.8/site-packages/oslo_privsep/daemon.py:371 2021-12-02 16:50:45.467 267 INFO oslo.privsep.daemon [-] privsep daemon starting 2021-12-02 16:50:45.471 267 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2021-12-02 16:50:45.474 267 ERROR oslo.privsep.daemon [-] [Errno 1] Operation not permitted Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 593, in helper_main Daemon(channel, context).run() File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 403, in run self._drop_privs() File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 439, in _drop_privs capabilities.drop_all_caps_except(self.caps, self.caps, []) File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/capabilities.py", line 156, in drop_all_caps_except raise OSError(errno, os.strerror(errno)) PermissionError: [Errno 1] Operation not permitted sudoers in the container appears to be set up correctly. Thanks Chris
Hi Chris, as for the issue itself - check if the container is running privileged (it should). As for the other details - Kolla-Ansible 12.x is for Wallaby, not Victoria, that could be causing issues. 12.0.0 is not the latest release for Wallaby either (12.2.0 is for today). Fixing these might help with the other issue. -yoctozepto On Thu, 2 Dec 2021 at 18:08, Chris DiLorenzo <cdilorenzo@gmail.com> wrote:
Hello,
I am running Kolla-12.0.0 with Openstack Victoria. I have mostly successfully deployed masakari. The instance monitoring works, I can set up segments in Horizon and via CLI. But I can't get masakari-hostmonitor to start:
2021-12-02 16:50:44.999 7 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'privsep-helper', '--config-file', '/etc/masakari-monitors/masakari -monitors.conf', '--privsep_context', 'masakarimonitors.privsep.monitors_priv', '--privsep_sock_path', '/tmp/tmpf_65hc7m/privsep.sock'] 2021-12-02 16:50:44.892 261 INFO oslo.privsep.daemon [-] privsep daemon starting 2021-12-02 16:50:44.898 261 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2021-12-02 16:50:44.902 261 ERROR oslo.privsep.daemon [-] [Errno 1] Operation not permitted Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 593, in helper_main Daemon(channel, context).run() File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 403, in run self._drop_privs() File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 439, in _drop_privs capabilities.drop_all_caps_except(self.caps, self.caps, []) File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/capabilities.py", line 156, in drop_all_caps_except raise OSError(errno, os.strerror(errno)) PermissionError: [Errno 1] Operation not permitted 2021-12-02 16:50:45.001 7 DEBUG oslo_privsep.comm [-] EOF on privsep read channel _reader_main /var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep /comm.py:149 2021-12-02 16:50:45.475 7 WARNING oslo.privsep.daemon [-] privsep log: [Errno 1] Operation not permitted 2021-12-02 16:50:45.565 7 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap 2021-12-02 16:50:45.565 7 DEBUG oslo.privsep.daemon [-] Accepted privsep connection to /tmp/tmpf_65hc7m/privsep.sock __init__ /var/lib/kolla/venv/lib/pytho n3.8/site-packages/oslo_privsep/daemon.py:371 2021-12-02 16:50:45.467 267 INFO oslo.privsep.daemon [-] privsep daemon starting 2021-12-02 16:50:45.471 267 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2021-12-02 16:50:45.474 267 ERROR oslo.privsep.daemon [-] [Errno 1] Operation not permitted Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 593, in helper_main Daemon(channel, context).run() File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 403, in run self._drop_privs() File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 439, in _drop_privs capabilities.drop_all_caps_except(self.caps, self.caps, []) File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/capabilities.py", line 156, in drop_all_caps_except raise OSError(errno, os.strerror(errno)) PermissionError: [Errno 1] Operation not permitted
sudoers in the container appears to be set up correctly.
Thanks Chris
It's not running it privileged mode: ubuntu@control001-poc:~$ docker inspect --format='{{.HostConfig.Privileged}}' masakari_hostmonitor false The config in ansible/roles/masakari/defaults/main.yml looks like this: masakari-hostmonitor: container_name: masakari_hostmonitor group: masakari-hostmonitor enabled: true ipc_mode: host image: "{{ masakari_monitors_image_full }}" volumes: "{{ masakari_hostmonitor_default_volumes + masakari_hostmonitor_extra_volumes }}" dimensions: "{{ masakari_hostmonitor_dimensions }}" The reason I am running 12.0.0 is that the hostmonitor code isn't in 11.x Thanks, Chris On Thu, Dec 2, 2021 at 2:28 PM Radosław Piliszek < radoslaw.piliszek@gmail.com> wrote:
Hi Chris,
as for the issue itself - check if the container is running privileged (it should).
As for the other details - Kolla-Ansible 12.x is for Wallaby, not Victoria, that could be causing issues. 12.0.0 is not the latest release for Wallaby either (12.2.0 is for today). Fixing these might help with the other issue.
-yoctozepto
On Thu, 2 Dec 2021 at 18:08, Chris DiLorenzo <cdilorenzo@gmail.com> wrote:
Hello,
I am running Kolla-12.0.0 with Openstack Victoria. I have mostly
successfully deployed masakari. The instance monitoring works, I can set up segments in Horizon and via CLI. But I can't get masakari-hostmonitor to start:
2021-12-02 16:50:44.999 7 INFO oslo.privsep.daemon [-] Running privsep
-monitors.conf', '--privsep_context', 'masakarimonitors.privsep.monitors_priv', '--privsep_sock_path', '/tmp/tmpf_65hc7m/privsep.sock'] 2021-12-02 16:50:44.892 261 INFO oslo.privsep.daemon [-] privsep daemon starting 2021-12-02 16:50:44.898 261 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2021-12-02 16:50:44.902 261 ERROR oslo.privsep.daemon [-] [Errno 1] Operation not permitted Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py",
helper: ['sudo', 'privsep-helper', '--config-file', '/etc/masakari-monitors/masakari line 593, in helper_main
Daemon(channel, context).run() File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 403, in run
self._drop_privs() File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 439, in _drop_privs
capabilities.drop_all_caps_except(self.caps, self.caps, []) File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/capabilities.py", line 156, in drop_all_caps_except
raise OSError(errno, os.strerror(errno)) PermissionError: [Errno 1] Operation not permitted 2021-12-02 16:50:45.001 7 DEBUG oslo_privsep.comm [-] EOF on privsep
/comm.py:149 2021-12-02 16:50:45.475 7 WARNING oslo.privsep.daemon [-] privsep log: [Errno 1] Operation not permitted 2021-12-02 16:50:45.565 7 INFO oslo.privsep.daemon [-] Spawned new
2021-12-02 16:50:45.565 7 DEBUG oslo.privsep.daemon [-] Accepted privsep connection to /tmp/tmpf_65hc7m/privsep.sock __init__ /var/lib/kolla/venv/lib/pytho n3.8/site-packages/oslo_privsep/daemon.py:371 2021-12-02 16:50:45.467 267 INFO oslo.privsep.daemon [-] privsep daemon starting 2021-12-02 16:50:45.471 267 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2021-12-02 16:50:45.474 267 ERROR oslo.privsep.daemon [-] [Errno 1] Operation not permitted Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py",
read channel _reader_main /var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep privsep daemon via rootwrap line 593, in helper_main
Daemon(channel, context).run() File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 403, in run
self._drop_privs() File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 439, in _drop_privs
capabilities.drop_all_caps_except(self.caps, self.caps, []) File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/capabilities.py", line 156, in drop_all_caps_except
raise OSError(errno, os.strerror(errno)) PermissionError: [Errno 1] Operation not permitted
sudoers in the container appears to be set up correctly.
Thanks Chris
Ah, sorry for the confusion. It's instancemonitor that should run privileged, not hostmonitor. That's right, hostmonitor is really supported only since Wallaby because that's when it was patched to run in containerised deployments. I suggest you simply upgrade your stack to Wallaby and enjoy it being supported. -yoctozepto On Thu, 2 Dec 2021 at 21:18, Chris DiLorenzo <cdilorenzo@gmail.com> wrote:
It's not running it privileged mode:
ubuntu@control001-poc:~$ docker inspect --format='{{.HostConfig.Privileged}}' masakari_hostmonitor false
The config in ansible/roles/masakari/defaults/main.yml looks like this:
masakari-hostmonitor: container_name: masakari_hostmonitor group: masakari-hostmonitor enabled: true ipc_mode: host image: "{{ masakari_monitors_image_full }}" volumes: "{{ masakari_hostmonitor_default_volumes + masakari_hostmonitor_extra_volumes }}" dimensions: "{{ masakari_hostmonitor_dimensions }}"
The reason I am running 12.0.0 is that the hostmonitor code isn't in 11.x
Thanks, Chris
On Thu, Dec 2, 2021 at 2:28 PM Radosław Piliszek < radoslaw.piliszek@gmail.com> wrote:
Hi Chris,
as for the issue itself - check if the container is running privileged (it should).
As for the other details - Kolla-Ansible 12.x is for Wallaby, not Victoria, that could be causing issues. 12.0.0 is not the latest release for Wallaby either (12.2.0 is for today). Fixing these might help with the other issue.
-yoctozepto
On Thu, 2 Dec 2021 at 18:08, Chris DiLorenzo <cdilorenzo@gmail.com> wrote:
Hello,
I am running Kolla-12.0.0 with Openstack Victoria. I have mostly
successfully deployed masakari. The instance monitoring works, I can set up segments in Horizon and via CLI. But I can't get masakari-hostmonitor to start:
2021-12-02 16:50:44.999 7 INFO oslo.privsep.daemon [-] Running privsep
-monitors.conf', '--privsep_context', 'masakarimonitors.privsep.monitors_priv', '--privsep_sock_path', '/tmp/tmpf_65hc7m/privsep.sock'] 2021-12-02 16:50:44.892 261 INFO oslo.privsep.daemon [-] privsep daemon starting 2021-12-02 16:50:44.898 261 INFO oslo.privsep.daemon [-] privsep
2021-12-02 16:50:44.902 261 ERROR oslo.privsep.daemon [-] [Errno 1] Operation not permitted Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py",
helper: ['sudo', 'privsep-helper', '--config-file', '/etc/masakari-monitors/masakari process running with uid/gid: 0/0 line 593, in helper_main
Daemon(channel, context).run() File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 403, in run
self._drop_privs() File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 439, in _drop_privs
capabilities.drop_all_caps_except(self.caps, self.caps, []) File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/capabilities.py", line 156, in drop_all_caps_except
raise OSError(errno, os.strerror(errno)) PermissionError: [Errno 1] Operation not permitted 2021-12-02 16:50:45.001 7 DEBUG oslo_privsep.comm [-] EOF on privsep
/comm.py:149 2021-12-02 16:50:45.475 7 WARNING oslo.privsep.daemon [-] privsep log: [Errno 1] Operation not permitted 2021-12-02 16:50:45.565 7 INFO oslo.privsep.daemon [-] Spawned new
2021-12-02 16:50:45.565 7 DEBUG oslo.privsep.daemon [-] Accepted
n3.8/site-packages/oslo_privsep/daemon.py:371 2021-12-02 16:50:45.467 267 INFO oslo.privsep.daemon [-] privsep daemon starting 2021-12-02 16:50:45.471 267 INFO oslo.privsep.daemon [-] privsep
2021-12-02 16:50:45.474 267 ERROR oslo.privsep.daemon [-] [Errno 1] Operation not permitted Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py",
read channel _reader_main /var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep privsep daemon via rootwrap privsep connection to /tmp/tmpf_65hc7m/privsep.sock __init__ /var/lib/kolla/venv/lib/pytho process running with uid/gid: 0/0 line 593, in helper_main
Daemon(channel, context).run() File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 403, in run
self._drop_privs() File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 439, in _drop_privs
capabilities.drop_all_caps_except(self.caps, self.caps, []) File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/capabilities.py", line 156, in drop_all_caps_except
raise OSError(errno, os.strerror(errno)) PermissionError: [Errno 1] Operation not permitted
sudoers in the container appears to be set up correctly.
Thanks Chris
participants (2)
-
Chris DiLorenzo
-
Radosław Piliszek