Open Stack

Hello!

Periodically we get "UNCAUGHT EXCEPTION#012Traceback" errors on
swift-proxy, log attached to this email. After that the swift-proxy process
crashes, clients get 502 errors. Could you please help with this? Swift
version - yoga 2.29.1.

Thank you. Regards.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20221103/762afed0/attachment-0001.htm>
-------------- next part --------------
proxy-server[2656473]: STDERR: Traceback (most recent call last):
proxy-server[2656473]: STDERR: File "/usr/lib/python3/dist-packages/eventlet/hubs/poll.py", line 111, in wait#012    listener.cb(fileno)
proxy-server[2656473]: STDERR: File "/usr/lib/python3/dist-packages/eventlet/greenthread.py", line 221, in main#012    result = function(*args, **kwargs)
proxy-server[2656473]: STDERR: File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 825, in process_request#012    proto.__init__(conn_state, self)
proxy-server[2656473]: STDERR: File "/usr/lib/python3/dist-packages/swift/common/wsgi.py", line 395, in __init__#012    wsgi.HttpProtocol.__init__(self, *args, **kwargs)
proxy-server[2656473]: STDERR: File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 357, in __init__#012    self.handle()
proxy-server[2656473]: STDERR: File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 390, in handle#012    self.handle_one_request()
proxy-server[2656473]: STDERR: File "/usr/lib/python3/dist-packages/swift/common/wsgi.py", line 521, in handle_one_request#012    got = wsgi.HttpProtocol.handle_one_request(self)
proxy-server[2656473]: STDERR: File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 419, in handle_one_request#012    self.raw_requestline = self._read_request_line()
proxy-server[2656473]: STDERR: File "/usr/lib/python3/dist-packages/swift/common/wsgi.py", line 513, in _read_request_line#012    got = wsgi.HttpProtocol._read_request_line(self)
proxy-server[2656473]: STDERR: File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 402, in _read_request_line#012    return self.rfile.readline(self.server.url_length_limit)
proxy-server[2656473]: STDERR: File "/usr/lib/python3.8/socket.py", line 669, in readinto#012    return self._sock.recv_into(b)
proxy-server[2656473]: STDERR: File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 374, in recv_into#012    return self._recv_loop(self.fd.recv_into, 0, buffer, nbytes, flags)
proxy-server[2656473]: STDERR: File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 352, in _recv_loop#012    return recv_meth(*args)
proxy-server[2656473]: STDERR: TimeoutError: [Errno 110] Connection timed out
proxy-server[2656473]: STDERR: Removing descriptor: 66
proxy-server[2657864]: UNCAUGHT EXCEPTION#012Traceback (most recent call last):#012  File "/usr/lib/python3/dist-packages/eventlet/hubs/poll.py", line 111, in wait#012    listener.cb(fileno)#012  File
 "/usr/lib/python3/dist-packages/eventlet/greenthread.py", line 221, in main#012    result = function(*args, **kwargs)#012  File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 825, in process_request#012    proto.__init__(conn
_state, self)#012  File "/usr/lib/python3/dist-packages/swift/common/wsgi.py", line 395, in __init__#012    wsgi.HttpProtocol.__init__(self, *args, **kwargs)#012  File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 357, in __i
nit__#012    self.handle()#012  File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 390, in handle#012    self.handle_one_request()#012  File "/usr/lib/python3/dist-packages/swift/common/wsgi.py", line 521, in handle_one_reque
st#012    got = wsgi.HttpProtocol.handle_one_request(self)#012  File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 419, in handle_one_request#012    self.raw_requestline = self._read_request_line()#012  File "/usr/lib/python3
/dist-packages/swift/common/wsgi.py", line 513, in _read_request_line#012    got = wsgi.HttpProtocol._read_request_line(self)#012  File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 402, in _read_request_line#012    return se
lf.rfile.readline(self.server.url_length_limit)#012  File "/usr/lib/python3.8/socket.py", line 669, in readinto#012    return self._sock.recv_into(b)#012  File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 374, in rec
v_into#012    return self._recv_loop(self.fd.recv_into, 0, buffer, nbytes, flags)#012  File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 352, in _recv_loop#012    return recv_meth(*args)#012TimeoutError: [Errno 110] 
Connection timed out#012#012During handling of the above exception, another exception occurred:#012#012Traceback (most recent call last):#012  File "/usr/lib/python3/dist-packages/swift/common/utils.py", line 6146, in acquire#012    os
.read(self.rfd, 1)#012BlockingIOError: [Errno 11] Resource temporarily unavailable#012#012During handling of the above exception, another exception occurred:#012#012Traceback (most recent call last):#012  File "/usr/bin/swift-proxy-ser
ver", line 23, in <module>#012    sys.exit(run_wsgi(conf_file, 'proxy-server', **options))#012  File "/usr/lib/python3/dist-packages/swift/common/wsgi.py", line 1108, in run_wsgi#012    run_server(conf, logger, sock, ready_callback=not
ify,#012  File "/usr/lib/python3/dist-packages/swift/common/wsgi.py", line 658, in run_server#012    wsgi.server(sock, app, wsgi_logger, **server_kwargs)#012  File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 990, in server#
012    client_socket, client_addr = sock.accept()#012  File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 233, in accept#012    self._trampoline(fd, read=True, timeout=self.gettimeout(), timeout_exc=_timeout_exc)#012 
 File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 211, in _trampoline#012    return trampoline(fd, read=read, write=write, timeout=timeout,#012  File "/usr/lib/python3/dist-packages/eventlet/hubs/__init__.py", line 
159, in trampoline#012    return hub.switch()#012  File "/usr/lib/python3/dist-packages/eventlet/hubs/hub.py", line 313, in switch#012    return self.greenlet.switch()#012  File "/usr/lib/python3/dist-packages/eventlet/hubs/hub.py", li
ne 365, in run#012    self.wait(sleep_time)#012  File "/usr/lib/python3/dist-packages/eventlet/hubs/poll.py", line 115, in wait#012    self.squelch_exception(fileno, sys.exc_info())#012  File "/usr/lib/python3/dist-packages/eventlet/hu
bs/hub.py", line 316, in squelch_exception#012    traceback.print_exception(*exc_info)#012  File "/usr/lib/python3.8/traceback.py", line 105, in print_exception#012    print(line, file=file, end="")#012  File "/usr/lib/python3/dist-packages/swift/common/utils.py", line 1837, in write#012    self.logger.error(_('%(type)s: %(value)s'),#012  File "/usr/lib/python3.8/logging/__init__.py", line 1823, in error#012    self.log(ERROR, msg, *args, **kwargs)#012  File "/usr/lib/python3.8/logging/__init__.py", line 1844, in log#012    self.logger.log(level, msg, *args, **kwargs)#012  File "/usr/lib/python3.8/logging/__init__.py", line 1512, in log#012    self._log(level, msg, args, **kwargs)#012  File "/usr/lib/python3.8/logging/__init__.py", line 1589, in _log#012    self.handle(record)#012  File "/usr/lib/python3.8/logging/__init__.py", line 1599, in handle#012    self.callHandlers(record)#012  File "/usr/lib/python3.8/logging/__init__.py", line 1661, in callHandlers#012    hdlr.handle(record)#012  File "/usr/lib/python3.8/logging/__init__.py", line 952, in handle#012    self.acquire()#012  File "/usr/lib/python3.8/logging/__init__.py", line 903, in acquire#012    self.lock.acquire()#012  File "/usr/lib/python3/dist-packages/swift/common/utils.py", line 6159, in acquire#012    eventlet.hubs.trampoline(self.rfd, read=True)#012  File "/usr/lib/python3/dist-packages/eventlet/hubs/__init__.py", line 141, in trampoline#012    assert hub.greenlet is not current, 'do not call blocking functions from the mainloop'#012AssertionError: do not call blocking functions from the mainloop
proxy-server[2603325]: Removing dead child 2657864 from parent 2603325
proxy-server[2657952]: STDERR: The option "bind_port" is not known to keystonemiddleware
proxy-server[2657952]: STDERR: The option "workers" is not known to keystonemiddleware
proxy-server[2657952]: STDERR: The option "user" is not known to keystonemiddleware
proxy-server[2657952]: STDERR: The option "log_level" is not known to keystonemiddleware
proxy-server[2657952]: STDERR: The option "log_facility" is not known to keystonemiddleware
proxy-server[2657952]: STDERR: The option "max_clients" is not known to keystonemiddleware
proxy-server[2657952]: STDERR: The option "client_timeout" is not known to keystonemiddleware
proxy-server[2657952]: STDERR: The option "recheck_account_existence" is not known to keystonemiddleware
proxy-server[2657952]: STDERR: The option "recheck_container_existence" is not known to keystonemiddleware
proxy-server[2657952]: STDERR: The option "log_name" is not known to keystonemiddleware
proxy-server[2657952]: STDERR: The option "auth_url" is not known to keystonemiddleware
proxy-server[2657952]: STDERR: The option "project_domain_id" is not known to keystonemiddleware
proxy-server[2657952]: STDERR: The option "user_domain_id" is not known to keystonemiddleware
proxy-server[2657952]: STDERR: The option "project_name" is not known to keystonemiddleware
proxy-server[2657952]: STDERR: The option "username" is not known to keystonemiddleware
proxy-server[2657952]: STDERR: The option "password" is not known to keystonemiddleware
proxy-server[2657952]: STDERR: The option "__name__" is not known to keystonemiddleware