Open Stack

Hey all,

We have some python stacktrace in our nova-compute journalctl logs,
which looks like that:

$ journalctl -u nova-compute
...
Nov 01 09:04:03 host1234 python3[161354]: Exception in thread tpool_thread_6:
Nov 01 09:04:03 host1234 python3[161354]: Traceback (most recent call last):
Nov 01 09:04:03 host1234 python3[161354]:   File "/opt/openstack/nova/lib/python3.6/threading.py", line 916, in _bootstrap_inner
Nov 01 09:04:03 host1234 python3[161354]:     self.run()
Nov 01 09:04:03 host1234 python3[161354]:   File "/opt/openstack/nova/lib/python3.6/threading.py", line 864, in run
Nov 01 09:04:03 host1234 python3[161354]:     self._target(*self._args, **self._kwargs)
Nov 01 09:04:03 host1234 python3[161354]:   File "/opt/openstack/nova/lib/python3.6/site-packages/eventlet/tpool.py", line 96, in tworker
Nov 01 09:04:03 host1234 python3[161354]:     _wsock.sendall(_bytetosend)
Nov 01 09:04:03 host1234 python3[161354]: TimeoutError: [Errno 110] Connection timed out
Nov 01 09:04:03 host1234 python3[161354]: Traceback (most recent call last):
Nov 01 09:04:03 host1234 python3[161354]:   File "/opt/openstack/nova/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 109, in wait
Nov 01 09:04:03 host1234 python3[161354]:     listener.cb(fileno)
Nov 01 09:04:03 host1234 python3[161354]:   File "/opt/openstack/nova/lib/python3.6/site-packages/eventlet/tpool.py", line 58, in tpool_trampoline
Nov 01 09:04:03 host1234 python3[161354]:     assert _c
Nov 01 09:04:03 host1234 python3[161354]: AssertionError
Nov 01 09:04:03 host1234 python3[161354]: Removing descriptor: 16
...


After this, nova-compute is "stuck" and does nothing more, but still
continue to answer on RPC (it is still up in nova services)

We am not experts of python threading / eventlet stuff, and we have no
idea how to debug this.

Our current solution is to restart nova-compute, but it's more a dirty
workaround than a real fix.

Does it ring a bell to someone in the community?

Cheers,
Arnaud