[oslo] oslo-rootwrap daemon test failures
There are a handful of failures when running the oslo-rootwrap test suite on FreeBSD. Most I have been able to fix, but I'm looking for some help understanding what is going wrong with the daemon tests. One example of a failing test is test_daemon_cleanup_client. The test reports: AssertionError: -9 == -9 : Server haven't stopped in one second which I _think_ occurs because of an exception while closing the connection (see below). Any suggestions as to what I might check (and where) to understand why this is happening? Thanks!
Running the latest from git on FreeBSD 13.2 with Python 3.9
{2} oslo_rootwrap.tests.test_functional.RootwrapDaemonTest.test_daemon_cleanup_client [1.455829s] ... FAILED
Captured traceback: ~~~~~~~~~~~~~~~~~~~ Traceback (most recent call last):
File "/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/oslo_rootwrap/tests/test_functional.py", line 299, in test_daemon_cleanup_client self.client._finalize()
File "/usr/local/lib/python3.9/contextlib.py", line 126, in __exit__ next(self.gen)
File "/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/oslo_rootwrap/tests/test_functional.py", line 289, in _test_daemon_cleanup self.assertNotEqual(-signal.SIGKILL, process.returncode,
File "/usr/local/lib/python3.9/unittest/case.py", line 846, in assertNotEqual raise self.failureException(msg)
AssertionError: -9 == -9 : Server haven't stopped in one second
Captured daemon_log: ~~~~~~~~~~~~~~~~~~~~ 2023-10-19 18:33:10,718 | [ 1664]+DEBUG | Created temporary directory /tmp/rootwrap-snf7oiau 2023-10-19 18:33:10,718 | [ 1664]+DEBUG | Will listen on socket /tmp/rootwrap-snf7oiau/rootwrap.sock 2023-10-19 18:33:10,719 | [ 1664]+ INFO | Starting rootwrap daemon main loop 2023-10-19 18:33:11,719 | [ 1664]+ INFO | Got signal 2. Shutting down server 2023-10-19 18:33:11,719 | [ 1664]+DEBUG | Removing temporary directory /tmp/rootwrap-snf7oiau Traceback (most recent call last): File "/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/.tox/py3/lib/python3.9/site-packages/oslo_rootwrap/daemon.py", line 166, in daemon_start server.serve_forever() File "/usr/local/lib/python3.9/multiprocessing/managers.py", line 184, in serve_forever sys.exit(0) SystemExit: 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/oslo_rootwrap/tests/run_daemon.py", line 58, in <module> cmd.daemon() File "/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/.tox/py3/lib/python3.9/site-packages/oslo_rootwrap/cmd.py", line 62, in daemon return main(run_daemon=True) File "/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/.tox/py3/lib/python3.9/site-packages/oslo_rootwrap/cmd.py", line 133, in main daemon_mod.daemon_start(config, filters) File "/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/.tox/py3/lib/python3.9/site-packages/oslo_rootwrap/daemon.py", line 171, in daemon_start conn.close() File "/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/.tox/py3/lib/python3.9/site-packages/oslo_rootwrap/jsonrpc.py", line 92, in close self._socket.shutdown(socket.SHUT_RDWR) OSError: [Errno 57] Socket is not connected
Captured client_log: ~~~~~~~~~~~~~~~~~~~~ 2023-10-19 18:33:10,290 | [ 1610] DEBUG | Popen for ['/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/.tox/py3/bin/python', '/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/oslo_rootwrap/tests/run_daemon.py', '/tmp/tmpjri8_7we/rootwrap.conf'] command has been instantiated 2023-10-19 18:33:10,719 | [ 1610] INFO | Spawned new rootwrap daemon process with pid=1664 2023-10-19 18:33:10,728 | [ 1610] INFO | Stopping rootwrap daemon process with pid=1664 2023-10-19 18:33:11,733 | [ 1610] INFO | Rootwrap daemon process exit with status: -9
Hi Chuck,
I would not be surprised if oslo-rootwrap needed a few code changes to work correctly on FreeBSD... It's been designed with a bunch of Linux assumptions. I'm not familiar enough with FreeBSD signal/socket handling to help debug this one.
Note that oslo-rootwrap is being deprecated in favor of oslo-privsep (which may rely on even more Linux assumptions :/ )
Thierry
Chuck Tuffli wrote:
There are a handful of failures when running the oslo-rootwrap test suite on FreeBSD. Most I have been able to fix, but I'm looking for some help understanding what is going wrong with the daemon tests. One example of a failing test is test_daemon_cleanup_client. The test reports: AssertionError: -9 == -9 : Server haven't stopped in one second which I _think_ occurs because of an exception while closing the connection (see below). Any suggestions as to what I might check (and where) to understand why this is happening? Thanks!
Running the latest from git on FreeBSD 13.2 with Python 3.9
{2} oslo_rootwrap.tests.test_functional.RootwrapDaemonTest.test_daemon_cleanup_client [1.455829s] ... FAILED
Captured traceback:
Traceback (most recent call last): File "/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/oslo_rootwrap/tests/test_functional.py", line 299, in test_daemon_cleanup_client self.client._finalize() File "/usr/local/lib/python3.9/contextlib.py", line 126, in __exit__ next(self.gen) File "/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/oslo_rootwrap/tests/test_functional.py", line 289, in _test_daemon_cleanup self.assertNotEqual(-signal.SIGKILL, process.returncode, File "/usr/local/lib/python3.9/unittest/case.py", line 846, in assertNotEqual raise self.failureException(msg) AssertionError: -9 == -9 : Server haven't stopped in one second Captured daemon_log:2023-10-19 18:33:10,718 | [ 1664]+DEBUG | Created temporarydirectory /tmp/rootwrap-snf7oiau 2023-10-19 18:33:10,718 | [ 1664]+DEBUG | Will listen on socket /tmp/rootwrap-snf7oiau/rootwrap.sock 2023-10-19 18:33:10,719 | [ 1664]+ INFO | Starting rootwrap daemon main loop 2023-10-19 18:33:11,719 | [ 1664]+ INFO | Got signal 2. Shutting down server 2023-10-19 18:33:11,719 | [ 1664]+DEBUG | Removing temporary directory /tmp/rootwrap-snf7oiau Traceback (most recent call last): File "/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/.tox/py3/lib/python3.9/site-packages/oslo_rootwrap/daemon.py", line 166, in daemon_start server.serve_forever() File "/usr/local/lib/python3.9/multiprocessing/managers.py", line 184, in serve_forever sys.exit(0) SystemExit: 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/oslo_rootwrap/tests/run_daemon.py", line 58, in <module> cmd.daemon() File "/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/.tox/py3/lib/python3.9/site-packages/oslo_rootwrap/cmd.py", line 62, in daemon return main(run_daemon=True) File "/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/.tox/py3/lib/python3.9/site-packages/oslo_rootwrap/cmd.py", line 133, in main daemon_mod.daemon_start(config, filters) File "/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/.tox/py3/lib/python3.9/site-packages/oslo_rootwrap/daemon.py", line 171, in daemon_start conn.close() File "/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/.tox/py3/lib/python3.9/site-packages/oslo_rootwrap/jsonrpc.py", line 92, in close self._socket.shutdown(socket.SHUT_RDWR) OSError: [Errno 57] Socket is not connected
Captured client_log:
2023-10-19 18:33:10,290 | [ 1610] DEBUG | Popen for ['/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/.tox/py3/bin/python', '/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/oslo_rootwrap/tests/run_daemon.py', '/tmp/tmpjri8_7we/rootwrap.conf'] command has been instantiated 2023-10-19 18:33:10,719 | [ 1610] INFO | Spawned new rootwrap daemon process with pid=1664 2023-10-19 18:33:10,728 | [ 1610] INFO | Stopping rootwrap daemon process with pid=1664 2023-10-19 18:33:11,733 | [ 1610] INFO | Rootwrap daemon process exit with status: -9
I agree with Thierry. This kind of issue can be related to something specific to a distros or even an OS in your case. I recently faced an issue in RHEL with unit tests due to the fact that the `cat ` command is installed by the `coreutils` which lead us to a test error => https://review.opendev.org/c/openstack/oslo.rootwrap/+/896492
I'd suggest you dig toward that.
Le ven. 20 oct. 2023 à 10:03, Thierry Carrez thierry@openstack.org a écrit :
Hi Chuck,
I would not be surprised if oslo-rootwrap needed a few code changes to work correctly on FreeBSD... It's been designed with a bunch of Linux assumptions. I'm not familiar enough with FreeBSD signal/socket handling to help debug this one.
Note that oslo-rootwrap is being deprecated in favor of oslo-privsep (which may rely on even more Linux assumptions :/ )
Thierry
Chuck Tuffli wrote:
There are a handful of failures when running the oslo-rootwrap test suite on FreeBSD. Most I have been able to fix, but I'm looking for some help understanding what is going wrong with the daemon tests. One example of a failing test is test_daemon_cleanup_client. The test reports: AssertionError: -9 == -9 : Server haven't stopped in one second which I _think_ occurs because of an exception while closing the connection (see below). Any suggestions as to what I might check (and where) to understand why this is happening? Thanks!
Running the latest from git on FreeBSD 13.2 with Python 3.9
{2}
oslo_rootwrap.tests.test_functional.RootwrapDaemonTest.test_daemon_cleanup_client
[1.455829s] ... FAILED
Captured traceback:
Traceback (most recent call last): File"/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/oslo_rootwrap/tests/test_functional.py",
line 299, in test_daemon_cleanup_client self.client._finalize()
File "/usr/local/lib/python3.9/contextlib.py", line 126, in__exit__
next(self.gen) File"/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/oslo_rootwrap/tests/test_functional.py",
line 289, in _test_daemon_cleanup self.assertNotEqual(-signal.SIGKILL, process.returncode,
File "/usr/local/lib/python3.9/unittest/case.py", line 846, inassertNotEqual raise self.failureException(msg)
AssertionError: -9 == -9 : Server haven't stopped in one secondCaptured daemon_log:
2023-10-19 18:33:10,718 | [ 1664]+DEBUG | Created temporary directory /tmp/rootwrap-snf7oiau 2023-10-19 18:33:10,718 | [ 1664]+DEBUG | Will listen on socket /tmp/rootwrap-snf7oiau/rootwrap.sock 2023-10-19 18:33:10,719 | [ 1664]+ INFO | Starting rootwrap daemon mainloop
2023-10-19 18:33:11,719 | [ 1664]+ INFO | Got signal 2. Shutting down
server
2023-10-19 18:33:11,719 | [ 1664]+DEBUG | Removing temporary directory /tmp/rootwrap-snf7oiau Traceback (most recent call last): File
"/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/.tox/py3/lib/python3.9/site-packages/oslo_rootwrap/daemon.py",
line 166, in daemon_start server.serve_forever() File "/usr/local/lib/python3.9/multiprocessing/managers.py", line 184, in serve_forever sys.exit(0) SystemExit: 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/oslo_rootwrap/tests/run_daemon.py",
line 58, in <module> cmd.daemon() File
"/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/.tox/py3/lib/python3.9/site-packages/oslo_rootwrap/cmd.py",
line 62, in daemon return main(run_daemon=True) File
"/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/.tox/py3/lib/python3.9/site-packages/oslo_rootwrap/cmd.py",
line 133, in main daemon_mod.daemon_start(config, filters) File
"/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/.tox/py3/lib/python3.9/site-packages/oslo_rootwrap/daemon.py",
line 171, in daemon_start conn.close() File
"/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/.tox/py3/lib/python3.9/site-packages/oslo_rootwrap/jsonrpc.py",
line 92, in close self._socket.shutdown(socket.SHUT_RDWR) OSError: [Errno 57] Socket is not connected
Captured client_log:
2023-10-19 18:33:10,290 | [ 1610] DEBUG | Popen for ['/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/.tox/py3/bin/python','/usr/home/tuffli/dev/openstack/oslo.rootwrap.git/oslo_rootwrap/tests/run_daemon.py',
'/tmp/tmpjri8_7we/rootwrap.conf'] command has been instantiated 2023-10-19 18:33:10,719 | [ 1610] INFO | Spawned new rootwrap daemon process with pid=1664 2023-10-19 18:33:10,728 | [ 1610] INFO | Stopping rootwrap daemon process with pid=1664 2023-10-19 18:33:11,733 | [ 1610] INFO | Rootwrap daemon process exit with status: -9
On Fri, Oct 20, 2023 at 1:00 AM Thierry Carrez thierry@openstack.org wrote:
Hi Chuck,
I would not be surprised if oslo-rootwrap needed a few code changes to work correctly on FreeBSD... It's been designed with a bunch of Linux assumptions. I'm not familiar enough with FreeBSD signal/socket handling to help debug this one.
Note that oslo-rootwrap is being deprecated in favor of oslo-privsep (which may rely on even more Linux assumptions :/ )
Thanks for the feedback, Thierry.
I took a peek at the oslo-privsep specification and code. Any thoughts as to what it would look like for non-Linux operating systems? E.g., emulation of Linux CAPs, genericized capabilities, something else?
If privsep is the path forward, I'd be interested in contributing support to it for FreeBSD, but a shove in the right direction would be appreciated :)
--chuck
Chuck Tuffli wrote:
On Fri, Oct 20, 2023 at 1:00 AM Thierry Carrez thierry@openstack.org wrote:
Hi Chuck,
I would not be surprised if oslo-rootwrap needed a few code changes to work correctly on FreeBSD... It's been designed with a bunch of Linux assumptions. I'm not familiar enough with FreeBSD signal/socket handling to help debug this one.
Note that oslo-rootwrap is being deprecated in favor of oslo-privsep (which may rely on even more Linux assumptions :/ )
Thanks for the feedback, Thierry.
I took a peek at the oslo-privsep specification and code. Any thoughts as to what it would look like for non-Linux operating systems? E.g., emulation of Linux CAPs, genericized capabilities, something else?
If privsep is the path forward, I'd be interested in contributing support to it for FreeBSD, but a shove in the right direction would be appreciated :)
I'm not sure which features in FreeBSD could be best leveraged to provide the same style of security layer than the capabilities-based approach that the Linux implementation follows... Maybe someone else on this list has an idea.
What's sure is that privsep is the future, and rootwrap is the past (that old design had several limitations and did not result in significantly increased security). So if you plan to spend some time on this, looking into porting privsep is probably the best use of your time.
Cheers,
participants (3)
-
Chuck Tuffli
-
Herve Beraud
-
Thierry Carrez