Hi Tony,
Well, it looks like memcached just timed out. I'd check the load on it.
-yoctozepto
On Thu, Sep 10, 2020 at 7:24 PM Tony Liu tonyliu0592@hotmail.com wrote:
Any clues on this timeout exception?
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context [req-534d9855-8113-450d-8f9f-d93c0d961d24 113ee63a9ed0466794e24d069efc302c 4c142a681d884010ab36a7ac687d910c - default default] timed out: socket.timeout: timed out 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context Traceback (most recent call last): 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/middleware/auth_context.py", line 103, in _inner 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context return method(self, request) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/middleware/auth_context.py", line 353, in process_request 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context resp = super(AuthContextMiddleware, self).process_request(request) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 411, in process_request 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context allow_expired=allow_expired) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 445, in _do_fetch_token 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context data = self.fetch_token(token, **kwargs) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/middleware/auth_context.py", line 248, in fetch_token 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context token, access_rules_support=ACCESS_RULES_MIN_VERSION) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystone/common/manager.py", line 115, in wrapped 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context __ret_val = __f(*args, **kwargs) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystone/token/provider.py", line 145, in validate_token 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context token = self._validate_token(token_id) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-26>", line 2, in _validate_token 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 1360, in get_or_create_for_user_func 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context key, user_func, timeout, should_cache_fn, (arg, kw) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 962, in get_or_create 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context async_creator, 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 187, in __enter__ 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context return self._enter() 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 94, in _enter 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context generated = self._enter_create(value, createdtime) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 180, in _enter_create 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context return self.creator() 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 916, in gen_value 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context *creator_args[0], **creator_args[1] 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystone/token/provider.py", line 179, in _validate_token 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context token.mint(token_id, issued_at) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystone/models/token_model.py", line 579, in mint 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context self._validate_token_resources() 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystone/models/token_model.py", line 471, in _validate_token_resources 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context if self.project and not self.project_domain.get('enabled'): 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystone/models/token_model.py", line 176, in project_domain 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context self.project['domain_id'] 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystone/common/manager.py", line 115, in wrapped 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context __ret_val = __f(*args, **kwargs) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-27>", line 2, in get_domain 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 1360, in get_or_create_for_user_func 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context key, user_func, timeout, should_cache_fn, (arg, kw) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 962, in get_or_create 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context async_creator, 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 187, in __enter__ 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context return self._enter() 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 87, in _enter 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context value = value_fn() 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 902, in get_value 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context value = self.backend.get(key) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystone/common/cache/_context_cache.py", line 74, in get 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context value = self.proxied.get(key) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/cache/backends/memcached.py", line 168, in get 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context value = self.client.get(key) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/oslo_cache/backends/memcache_pool.py", line 32, in _run_method 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context return getattr(client, __name)(*args, **kwargs) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/memcache.py", line 1129, in get 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context return self._get('get', key) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/memcache.py", line 1074, in _get 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context server, key = self._get_server(key) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/memcache.py", line 446, in _get_server 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context if server.connect(): 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/memcache.py", line 1391, in connect 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context if self._get_socket(): 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/memcache.py", line 1423, in _get_socket 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context self.flush() 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/memcache.py", line 1498, in flush 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context self.expect(b'OK') 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/memcache.py", line 1473, in expect 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context line = self.readline(raise_exception) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/memcache.py", line 1459, in readline 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context data = recv(4096) 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context socket.timeout: timed out
Thanks! Tony