Hi, I am using OpenStack Keystone Stein and run into the below error often where Keystone public process(listening to 5000) is running inside Apache httpd runs into the below. This problem is resolved with a restart of httpd service. Has anyone run into a similar issue ? This is seen soon after httpd is restarted and does not happen all the time. My environment has MariaDB backend. This problem is not limited to the assignment table and is seen across all other tables in Keystone. MariaDB service is functional and all the tables are in place. [Fri Aug 07 08:20:59.936087 2020] [:info] [pid 1420287] mod_wsgi (pid=1420287, process='keystone-public', application=''): Loading WSGI script '/usr/bin/keystone-wsgi-public'. [Fri Aug 07 08:20:59.936089 2020] [:info] [pid 1420288] mod_wsgi (pid=1420288, process='keystone-admin', application=''): Loading WSGI script '/usr/bin/keystone-wsgi-admin'. [Fri Aug 07 08:20:59.943431 2020] [ssl:info] [pid 1420290] [client 1.2.3.95:35762] AH01964: Connection to child 1 established (server 1.2.3.50:5000) [Fri Aug 07 08:21:00.009317 2020] [ssl:info] [pid 1420291] [client 1.2.3.113:60132] AH01964: Connection to child 2 established (server 1.2.3.50:5000) [Fri Aug 07 08:21:01.243594 2020] [ssl:info] [pid 1420289] [client 1.2.3.50:53996] AH01964: Connection to child 0 established (server 1.2.3.50:5000) [Fri Aug 07 08:21:01.386329 2020] [ssl:info] [pid 1420293] [client x.x.x.x:38645] AH01964: Connection to child 4 established (server 1.2.3.50:5000) [Fri Aug 07 08:21:01.824041 2020] [ssl:info] [pid 1420349] [client 1.2.3.101:42974] AH01964: Connection to child 5 established (server 1.2.3.50:5000) [Fri Aug 07 08:21:02.949166 2020] [ssl:info] [pid 1420378] [client 1.2.3.50:54014] AH01964: Connection to child 9 established (server 1.2.3.50:5000) [Fri Aug 07 08:21:02.949172 2020] [ssl:info] [pid 1420379] [client 1.2.3.80:46924] AH01964: Connection to child 10 established (server 1.2.3.50:5000) [Fri Aug 07 08:21:03.286057 2020] [:info] [pid 1420287] mod_wsgi (pid=1420287): Create interpreter '1.2.3.50:5000|'. [Fri Aug 07 08:21:03.287286 2020] [:info] [pid 1420287] [remote 1.2.3.95:156] mod_wsgi (pid=1420287, process='keystone-public', application='1.2.3.50:5000|'): Loading WSGI script '/usr/bin/keystone-wsgi-public'. [Fri Aug 07 08:21:04.675059 2020] [ssl:info] [pid 1420436] [client 1.2.3.50:54032] AH01964: Connection to child 12 established (server 1.2.3.50:5000) [Fri Aug 07 08:21:04.705975 2020] [ssl:info] [pid 1420437] [client 1.2.3.107:59554] AH01964: Connection to child 13 established (server 1.2.3.50:5000) [Fri Aug 07 08:21:06.960940 2020] [ssl:info] [pid 1420438] [client 1.2.3.80:46970] AH01964: Connection to child 14 established (server 1.2.3.50:5000) [Fri Aug 07 08:21:07.661670 2020] [ssl:info] [pid 1420349] [client 1.2.3.50:54124] AH01964: Connection to child 5 established (server 1.2.3.50:5000) [Fri Aug 07 08:21:07.683383 2020] [ssl:info] [pid 1420292] [client x.x.x.x:30065] AH01964: Connection to child 3 established (server 1.2.3.50:5000) [Fri Aug 07 08:21:08.442956 2020] [:error] [pid 1420287] [remote x.x.x.x:144] mod_wsgi (pid=1420287): Exception occurred processing WSGI script '/usr/bin/keystone-wsgi-public'. [Fri Aug 07 08:21:08.443002 2020] [:error] [pid 1420287] [remote x.x.x.x:144] Traceback (most recent call last): [Fri Aug 07 08:21:08.443017 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/python2.7/site-packages/flask/app.py", line 2309, in __call__ [Fri Aug 07 08:21:08.443509 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self.wsgi_app(environ, start_response) [Fri Aug 07 08:21:08.443525 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/python2.7/site-packages/werkzeug/contrib/fixers.py", line 152, in __call__ [Fri Aug 07 08:21:08.443630 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self.app(environ, start_response) [Fri Aug 07 08:21:08.443644 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__ [Fri Aug 07 08:21:08.443746 2020] [:error] [pid 1420287] [remote x.x.x.x:144] resp = self.call_func(req, *args, **kw) [Fri Aug 07 08:21:08.443756 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func [Fri Aug 07 08:21:08.443773 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self.func(req, *args, **kwargs) [Fri Aug 07 08:21:08.443781 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/python2.7/site-packages/oslo_middleware/base.py", line 131, in __call__ [Fri Aug 07 08:21:08.443844 2020] [:error] [pid 1420287] [remote x.x.x.x:144] response = req.get_response(self.application) .. ... .... .. [Fri Aug 07 08:21:08.450055 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 1216, in creator [Fri Aug 07 08:21:08.450071 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return fn(*arg, **kw) [Fri Aug 07 08:21:08.450080 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/python2.7/site-packages/keystone/assignment/core.py", line 128, in get_roles_for_user_and_project [Fri Aug 07 08:21:08.450975 2020] [:error] [pid 1420287] [remote x.x.x.x:144] user_id=user_id, project_id=project_id, effective=True) [Fri Aug 07 08:21:08.450985 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/python2.7/site-packages/keystone/common/manager.py", line 116, in wrapped [Fri Aug 07 08:21:08.451001 2020] [:error] [pid 1420287] [remote x.x.x.x:144] __ret_val = __f(*args, **kwargs) [Fri Aug 07 08:21:08.451009 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/python2.7/site-packages/keystone/assignment/core.py", line 999, in list_role_assignments [Fri Aug 07 08:21:08.451025 2020] [:error] [pid 1420287] [remote x.x.x.x:144] strip_domain_roles) [Fri Aug 07 08:21:08.451033 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/python2.7/site-packages/keystone/assignment/core.py", line 845, in _list_effective_role_assignments [Fri Aug 07 08:21:08.451049 2020] [:error] [pid 1420287] [remote x.x.x.x:144] domain_id=domain_id, inherited=inherited) [Fri Aug 07 08:21:08.451057 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/python2.7/site-packages/keystone/assignment/core.py", line 780, in list_role_assignments_for_actor [Fri Aug 07 08:21:08.451072 2020] [:error] [pid 1420287] [remote x.x.x.x:144] group_ids=group_ids, inherited_to_projects=False) [Fri Aug 07 08:21:08.451081 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/python2.7/site-packages/keystone/assignment/backends/sql.py", line 248, in list_role_assignments [Fri Aug 07 08:21:08.451599 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return [denormalize_role(ref) for ref in query.all()] [Fri Aug 07 08:21:08.451609 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, in all [Fri Aug 07 08:21:08.451632 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return list(self) [Fri Aug 07 08:21:08.451641 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3081, in __iter__ [Fri Aug 07 08:21:08.451656 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self._execute_and_instances(context) [Fri Aug 07 08:21:08.451665 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3106, in _execute_and_instances [Fri Aug 07 08:21:08.451683 2020] [:error] [pid 1420287] [remote x.x.x.x:144] result = conn.execute(querycontext.statement, self._params) [Fri Aug 07 08:21:08.451691 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 980, in execute [Fri Aug 07 08:21:08.451711 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return meth(self, multiparams, params) [Fri Aug 07 08:21:08.451720 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection [Fri Aug 07 08:21:08.451736 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return connection._execute_clauseelement(self, multiparams, params) [Fri Aug 07 08:21:08.451745 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement [Fri Aug 07 08:21:08.451762 2020] [:error] [pid 1420287] [remote x.x.x.x:144] distilled_params, [Fri Aug 07 08:21:08.451771 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context [Fri Aug 07 08:21:08.451786 2020] [:error] [pid 1420287] [remote x.x.x.x:144] e, statement, parameters, cursor, context [Fri Aug 07 08:21:08.451795 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception [Fri Aug 07 08:21:08.451810 2020] [:error] [pid 1420287] [remote x.x.x.x:144] util.raise_from_cause(newraise, exc_info) [Fri Aug 07 08:21:08.451818 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause [Fri Aug 07 08:21:08.451834 2020] [:error] [pid 1420287] [remote x.x.x.x:144] reraise(type(exception), exception, tb=exc_tb, cause=cause) [Fri Aug 07 08:21:08.451843 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context [Fri Aug 07 08:21:08.451858 2020] [:error] [pid 1420287] [remote x.x.x.x:144] cursor, statement, parameters, context [Fri Aug 07 08:21:08.451866 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute [Fri Aug 07 08:21:08.451882 2020] [:error] [pid 1420287] [remote x.x.x.x:144] cursor.execute(statement, parameters) [Fri Aug 07 08:21:08.451923 2020] [:error] [pid 1420287] [remote x.x.x.x:144] DBNonExistentTable: (sqlite3.OperationalError) no such table: assignment [SQL: u'SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, assignment.inherited AS assignment_inherited \\nFROM assignment \\nWHERE assignment.actor_id IN (?) AND assignment.target_id IN (?) AND assignment.type IN (?) AND assignment.inherited = 0'] [parameters: ('15c2fe91e053af57a997c568c117c908d59c138f996bdc19ae97e9f16df12345', '12345978536e45ab8a279e2b0fa4f947', 'UserProject')] (Background on this error at: http://sqlalche.me/e/e3q8) Regards, Divya