Hey all,
We have some python stacktrace in our nova-compute journalctl logs, which looks like that:
$ journalctl -u nova-compute ... Nov 01 09:04:03 host1234 python3[161354]: Exception in thread tpool_thread_6: Nov 01 09:04:03 host1234 python3[161354]: Traceback (most recent call last): Nov 01 09:04:03 host1234 python3[161354]: File "/opt/openstack/nova/lib/python3.6/threading.py", line 916, in _bootstrap_inner Nov 01 09:04:03 host1234 python3[161354]: self.run() Nov 01 09:04:03 host1234 python3[161354]: File "/opt/openstack/nova/lib/python3.6/threading.py", line 864, in run Nov 01 09:04:03 host1234 python3[161354]: self._target(*self._args, **self._kwargs) Nov 01 09:04:03 host1234 python3[161354]: File "/opt/openstack/nova/lib/python3.6/site-packages/eventlet/tpool.py", line 96, in tworker Nov 01 09:04:03 host1234 python3[161354]: _wsock.sendall(_bytetosend) Nov 01 09:04:03 host1234 python3[161354]: TimeoutError: [Errno 110] Connection timed out Nov 01 09:04:03 host1234 python3[161354]: Traceback (most recent call last): Nov 01 09:04:03 host1234 python3[161354]: File "/opt/openstack/nova/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 109, in wait Nov 01 09:04:03 host1234 python3[161354]: listener.cb(fileno) Nov 01 09:04:03 host1234 python3[161354]: File "/opt/openstack/nova/lib/python3.6/site-packages/eventlet/tpool.py", line 58, in tpool_trampoline Nov 01 09:04:03 host1234 python3[161354]: assert _c Nov 01 09:04:03 host1234 python3[161354]: AssertionError Nov 01 09:04:03 host1234 python3[161354]: Removing descriptor: 16 ...
After this, nova-compute is "stuck" and does nothing more, but still continue to answer on RPC (it is still up in nova services)
We am not experts of python threading / eventlet stuff, and we have no idea how to debug this.
Our current solution is to restart nova-compute, but it's more a dirty workaround than a real fix.
Does it ring a bell to someone in the community?
Cheers, Arnaud