Open Stack

Hi Tony,

Well, it looks like memcached just timed out.
I'd check the load on it.

-yoctozepto

On Thu, Sep 10, 2020 at 7:24 PM Tony Liu <tonyliu0592 at hotmail.com> wrote:
>
> Any clues on this timeout exception?
>
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context [req-534d9855-8113-450d-8f9f-d93c0d961d24 113ee63a9ed0466794e24d069efc302c 4c142a681d884010ab36a7ac687d910c - default default] timed out: socket.timeout: timed out
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context Traceback (most recent call last):
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/middleware/auth_context.py", line 103, in _inner
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     return method(self, request)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/middleware/auth_context.py", line 353, in process_request
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     resp = super(AuthContextMiddleware, self).process_request(request)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 411, in process_request
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     allow_expired=allow_expired)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 445, in _do_fetch_token
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     data = self.fetch_token(token, **kwargs)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/middleware/auth_context.py", line 248, in fetch_token
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     token, access_rules_support=ACCESS_RULES_MIN_VERSION)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/common/manager.py", line 115, in wrapped
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     __ret_val = __f(*args, **kwargs)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/token/provider.py", line 145, in validate_token
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     token = self._validate_token(token_id)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-26>", line 2, in _validate_token
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 1360, in get_or_create_for_user_func
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     key, user_func, timeout, should_cache_fn, (arg, kw)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 962, in get_or_create
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     async_creator,
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 187, in __enter__
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     return self._enter()
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 94, in _enter
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     generated = self._enter_create(value, createdtime)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 180, in _enter_create
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     return self.creator()
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 916, in gen_value
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     *creator_args[0], **creator_args[1]
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/token/provider.py", line 179, in _validate_token
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     token.mint(token_id, issued_at)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/models/token_model.py", line 579, in mint
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     self._validate_token_resources()
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/models/token_model.py", line 471, in _validate_token_resources
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     if self.project and not self.project_domain.get('enabled'):
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/models/token_model.py", line 176, in project_domain
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     self.project['domain_id']
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/common/manager.py", line 115, in wrapped
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     __ret_val = __f(*args, **kwargs)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-27>", line 2, in get_domain
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 1360, in get_or_create_for_user_func
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     key, user_func, timeout, should_cache_fn, (arg, kw)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 962, in get_or_create
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     async_creator,
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 187, in __enter__
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     return self._enter()
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 87, in _enter
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     value = value_fn()
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 902, in get_value
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     value = self.backend.get(key)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/keystone/common/cache/_context_cache.py", line 74, in get
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     value = self.proxied.get(key)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/dogpile/cache/backends/memcached.py", line 168, in get
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     value = self.client.get(key)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/oslo_cache/backends/memcache_pool.py", line 32, in _run_method
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     return getattr(client, __name)(*args, **kwargs)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/memcache.py", line 1129, in get
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     return self._get('get', key)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/memcache.py", line 1074, in _get
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     server, key = self._get_server(key)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/memcache.py", line 446, in _get_server
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     if server.connect():
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/memcache.py", line 1391, in connect
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     if self._get_socket():
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/memcache.py", line 1423, in _get_socket
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     self.flush()
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/memcache.py", line 1498, in flush
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     self.expect(b'OK')
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/memcache.py", line 1473, in expect
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     line = self.readline(raise_exception)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context   File "/usr/lib/python3.6/site-packages/memcache.py", line 1459, in readline
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context     data = recv(4096)
> 2020-09-10 10:10:33.981 28 ERROR keystone.server.flask.request_processing.middleware.auth_context socket.timeout: timed out
>
>
> Thanks!
> Tony
>
>