memcached load is heavy. I started another thread to get clarifications. This is not Keystone specific issue.
Thanks! Tony
-----Original Message----- From: Radosław Piliszek radoslaw.piliszek@gmail.com Sent: Friday, September 11, 2020 12:08 AM To: Tony Liu tonyliu0592@hotmail.com Cc: openstack-discuss openstack-discuss@lists.openstack.org Subject: Re: [Keystone] socket.timeout: timed out
Hi Tony,
Well, it looks like memcached just timed out. I'd check the load on it.
-yoctozepto
On Thu, Sep 10, 2020 at 7:24 PM Tony Liu tonyliu0592@hotmail.com wrote:
Any clues on this timeout exception?
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context [req- 534d9855-8113-450d-8f9f-d93c0d961d24 113ee63a9ed0466794e24d069efc302c 4c142a681d884010ab36a7ac687d910c - default default] timed out: socket.timeout: timed out
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context Traceback (most recent call last):
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site- packages/keystone/server/flask/request_processing/middleware/auth_contex t.py", line 103, in _inner
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context return method(self, request)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site- packages/keystone/server/flask/request_processing/middleware/auth_contex t.py", line 353, in process_request
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context resp = super(AuthContextMiddleware, self).process_request(request)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site- packages/keystonemiddleware/auth_token/__init__.py", line 411, in process_request
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context allow_expired=allow_expired)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site- packages/keystonemiddleware/auth_token/__init__.py", line 445, in _do_fetch_token
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context data = self.fetch_token(token, **kwargs)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site- packages/keystone/server/flask/request_processing/middleware/auth_contex t.py", line 248, in fetch_token
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context token, access_rules_support=ACCESS_RULES_MIN_VERSION)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystone/common/manager.py", line 115, in wrapped
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context __ret_val = __f(*args, **kwargs)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystone/token/provider.py", line 145, in validate_token
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context token = self._validate_token(token_id)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-26>", line 2, in _validate_token
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 1360, in get_or_create_for_user_func
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context key, user_func, timeout, should_cache_fn, (arg, kw)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 962, in get_or_create
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context async_creator,
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 187, in __enter__
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context return self._enter()
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 94, in _enter
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context generated = self._enter_create(value, createdtime)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 180, in _enter_create
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context return self.creator()
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 916, in gen_value
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context *creator_args[0], **creator_args[1]
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystone/token/provider.py", line 179, in _validate_token
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context token.mint(token_id, issued_at)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystone/models/token_model.py", line 579, in mint
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context self._validate_token_resources()
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystone/models/token_model.py", line 471, in _validate_token_resources
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context if self.project and not self.project_domain.get('enabled'):
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystone/models/token_model.py", line 176, in project_domain
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context self.project['domain_id']
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/keystone/common/manager.py", line 115, in wrapped
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context __ret_val = __f(*args, **kwargs)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-27>", line 2, in get_domain
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 1360, in get_or_create_for_user_func
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context key, user_func, timeout, should_cache_fn, (arg, kw)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 962, in get_or_create
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context async_creator,
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 187, in __enter__
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context return self._enter()
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 87, in _enter
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context value = value_fn()
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 902, in get_value
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context value = self.backend.get(key)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site- packages/keystone/common/cache/_context_cache.py", line 74, in get
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context value = self.proxied.get(key)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/dogpile/cache/backends/memcached.py", line 168, in get
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context value = self.client.get(key)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/oslo_cache/backends/memcache_pool.py", line 32, in _run_method
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context return getattr(client, __name)(*args, **kwargs)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/memcache.py", line 1129, in get
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context return self._get('get', key)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/memcache.py", line 1074, in _get
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context server, key = self._get_server(key)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/memcache.py", line 446, in _get_server
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context if server.connect():
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/memcache.py", line 1391, in connect
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context if self._get_socket():
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/memcache.py", line 1423, in _get_socket
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context self.flush()
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/memcache.py", line 1498, in flush
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context self.expect(b'OK')
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/memcache.py", line 1473, in expect
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context line = self.readline(raise_exception)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context File "/usr/lib/python3.6/site-packages/memcache.py", line 1459, in readline
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context data = recv(4096)
2020-09-10 10:10:33.981 28 ERROR
keystone.server.flask.request_processing.middleware.auth_context socket.timeout: timed out
Thanks! Tony