Hi,
Suddenly we have observed few VM down . then we have found some agent are getting down (XXX) , agents are getting UP and down randomly. Please check the attachment.
//////////////////////////////////////////////////////////////////////////////////////////////////// /sqlalchemy/pool.py", line 788, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout\n rec = pool._do_get()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1186, in _do_get\n (self.size(), self.overflow(), self._timeout), code="3o7r")\n', u'TimeoutError: QueuePool limit of size 5 overflow 50 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)%5Cn']. 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 837, in _report_state 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent True) 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent return method(context, 'report_state', **kwargs) 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent retry=self.retry) 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent retry=retry) 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent call_monitor_timeout, retry=retry) 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent raise result 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent RemoteError: Remote error: TimeoutError QueuePool limit of size 5 overflow 50 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r) 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch\n
/////////////////////////////////////////////////////////////////////////////////
Is there anything related with the following default configuration.
/etc/neutron/neutron.conf #max_pool_size = 5 #max_overflow = 50
regards, Munna