Suddenly we have observed few VM down . then we have found some agent are getting down (XXX) , agents are getting UP and down randomly. Please check the attachment.
////////////////////////////////////////////////////////////////////////////////////////////////////
/sqlalchemy/pool.py", line 788, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout\n rec = pool._do_get()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1186, in _do_get\n (self.size(), self.overflow(), self._timeout), code="3o7r")\n', u'TimeoutError: QueuePool limit of size 5 overflow 50 reached, connection timed out, timeout 30 (Background on this error at:
http://sqlalche.me/e/3o7r)\n'].
2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 837, in _report_state
2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent True)
2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state
2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent retry=self.retry)
2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent retry=retry)
2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send
2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent call_monitor_timeout, retry=retry)
2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send
2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent raise result
2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent RemoteError: Remote error: TimeoutError QueuePool limit of size 5 overflow 50 reached, connection timed out, timeout 30 (Background on this error at:
http://sqlalche.me/e/3o7r)
2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch\n
/////////////////////////////////////////////////////////////////////////////////