Open Stack

memcached load is heavy. I started another thread to get
clarifications. This is not Keystone specific issue.

Thanks!
Tony
> -----Original Message-----
> From: Radosław Piliszek <radoslaw.piliszek at gmail.com>
> Sent: Friday, September 11, 2020 12:08 AM
> To: Tony Liu <tonyliu0592 at hotmail.com>
> Cc: openstack-discuss <openstack-discuss at lists.openstack.org>
> Subject: Re: [Keystone] socket.timeout: timed out
> 
> Hi Tony,
> 
> Well, it looks like memcached just timed out.
> I'd check the load on it.
> 
> -yoctozepto
> 
> On Thu, Sep 10, 2020 at 7:24 PM Tony Liu <tonyliu0592 at hotmail.com> wrote:
> >
> > Any clues on this timeout exception?
> >
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context [req-
> 534d9855-8113-450d-8f9f-d93c0d961d24 113ee63a9ed0466794e24d069efc302c
> 4c142a681d884010ab36a7ac687d910c - default default] timed out:
> socket.timeout: timed out
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> Traceback (most recent call last):
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-
> packages/keystone/server/flask/request_processing/middleware/auth_contex
> t.py", line 103, in _inner
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> return method(self, request)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-
> packages/keystone/server/flask/request_processing/middleware/auth_contex
> t.py", line 353, in process_request
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> resp = super(AuthContextMiddleware, self).process_request(request)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-
> packages/keystonemiddleware/auth_token/__init__.py", line 411, in
> process_request
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> allow_expired=allow_expired)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-
> packages/keystonemiddleware/auth_token/__init__.py", line 445, in
> _do_fetch_token
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> data = self.fetch_token(token, **kwargs)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-
> packages/keystone/server/flask/request_processing/middleware/auth_contex
> t.py", line 248, in fetch_token
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> token, access_rules_support=ACCESS_RULES_MIN_VERSION)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/keystone/common/manager.py", line 115,
> in wrapped
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> __ret_val = __f(*args, **kwargs)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/keystone/token/provider.py", line 145,
> in validate_token
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> token = self._validate_token(token_id)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-26>", line
> 2, in _validate_token
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 1360,
> in get_or_create_for_user_func
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context     key,
> user_func, timeout, should_cache_fn, (arg, kw)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 962, in
> get_or_create
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> async_creator,
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 187, in
> __enter__
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> return self._enter()
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 94, in _enter
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> generated = self._enter_create(value, createdtime)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 180, in
> _enter_create
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> return self.creator()
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 916, in
> gen_value
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> *creator_args[0], **creator_args[1]
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/keystone/token/provider.py", line 179,
> in _validate_token
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> token.mint(token_id, issued_at)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/keystone/models/token_model.py", line
> 579, in mint
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> self._validate_token_resources()
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/keystone/models/token_model.py", line
> 471, in _validate_token_resources
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context     if
> self.project and not self.project_domain.get('enabled'):
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/keystone/models/token_model.py", line
> 176, in project_domain
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> self.project['domain_id']
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/keystone/common/manager.py", line 115,
> in wrapped
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> __ret_val = __f(*args, **kwargs)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-27>", line
> 2, in get_domain
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 1360,
> in get_or_create_for_user_func
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context     key,
> user_func, timeout, should_cache_fn, (arg, kw)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 962, in
> get_or_create
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> async_creator,
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 187, in
> __enter__
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> return self._enter()
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 87, in _enter
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> value = value_fn()
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 902, in
> get_value
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> value = self.backend.get(key)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-
> packages/keystone/common/cache/_context_cache.py", line 74, in get
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> value = self.proxied.get(key)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/dogpile/cache/backends/memcached.py",
> line 168, in get
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> value = self.client.get(key)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/oslo_cache/backends/memcache_pool.py",
> line 32, in _run_method
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> return getattr(client, __name)(*args, **kwargs)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/memcache.py", line 1129, in get
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> return self._get('get', key)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/memcache.py", line 1074, in _get
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> server, key = self._get_server(key)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/memcache.py", line 446, in _get_server
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context     if
> server.connect():
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/memcache.py", line 1391, in connect
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context     if
> self._get_socket():
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/memcache.py", line 1423, in
> _get_socket
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> self.flush()
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/memcache.py", line 1498, in flush
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> self.expect(b'OK')
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/memcache.py", line 1473, in expect
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> line = self.readline(raise_exception)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context   File
> "/usr/lib/python3.6/site-packages/memcache.py", line 1459, in readline
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> data = recv(4096)
> > 2020-09-10 10:10:33.981 28 ERROR
> keystone.server.flask.request_processing.middleware.auth_context
> socket.timeout: timed out
> >
> >
> > Thanks!
> > Tony
> >
> >