Open Stack

Any clues on this timeout exception?

2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context [req-534d9855-8113-450d-8f9f-d93c0d961d24 113ee63a9ed0466794e24d069efc302c 4c142a681d884010ab36a7ac687d910c - default default] timed out: socket.timeout: timed out
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context Traceback (most recent call last):
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/middleware/auth_context.py", line 103, in _inner
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     return method(self, request)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/middleware/auth_context.py", line 353, in process_request
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     resp = super(AuthContextMiddleware, self).process_request(request)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 411, in process_request
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     allow_expired=allow_expired)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 445, in _do_fetch_token
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     data = self.fetch_token(token, **kwargs)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/middleware/auth_context.py", line 248, in fetch_token
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     token, access_rules_support=ACCESS_RULES_MIN_VERSION)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/common/manager.py", line 115, in wrapped
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     __ret_val = __f(*args, **kwargs)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/token/provider.py", line 145, in validate_token
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     token = self._validate_token(token_id)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-26>", line 2, in _validate_token
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 1360, in get_or_create_for_user_func
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     key, user_func, timeout, should_cache_fn, (arg, kw)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 962, in get_or_create
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     async_creator,  
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 187, in __enter__
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     return self._enter()
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 94, in _enter
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     generated = self._enter_create(value, createdtime)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 180, in _enter_create
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     return self.creator()
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 916, in gen_value
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     *creator_args[0], **creator_args[1]
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/token/provider.py", line 179, in _validate_token
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     token.mint(token_id, issued_at)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/models/token_model.py", line 579, in mint
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     self._validate_token_resources()
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/models/token_model.py", line 471, in _validate_token_resources
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     if self.project and not self.project_domain.get('enabled'):
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/models/token_model.py", line 176, in project_domain
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     self.project['domain_id']
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/common/manager.py", line 115, in wrapped
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     __ret_val = __f(*args, **kwargs)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-27>", line 2, in get_domain
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 1360, in get_or_create_for_user_func
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     key, user_func, timeout, should_cache_fn, (arg, kw)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 962, in get_or_create
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     async_creator,  
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 187, in __enter__
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     return self._enter()
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 87, in _enter
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     value = value_fn()
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 902, in get_value
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     value = self.backend.get(key)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/common/cache/_context_cache.py", line 74, in get
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     value = self.proxied.get(key)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/cache/backends/memcached.py", line 168, in get
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     value = self.client.get(key)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/oslo_cache/backends/memcache_pool.py", line 32, in _run_method
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     return getattr(client, __name)(*args, **kwargs)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/memcache.py", line 1129, in get
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     return self._get('get', key)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/memcache.py", line 1074, in _get
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     server, key = self._get_server(key)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/memcache.py", line 446, in _get_server
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     if server.connect():
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/memcache.py", line 1391, in connect
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     if self._get_socket():
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/memcache.py", line 1423, in _get_socket
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     self.flush()
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/memcache.py", line 1498, in flush
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     self.expect(b'OK')
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/memcache.py", line 1473, in expect
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     line = self.readline(raise_exception)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/memcache.py", line 1459, in readline
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     data = recv(4096)
2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context socket.timeout: timed out


Thanks!
Tony