Hi,

On poniedziałek, 24 stycznia 2022 09:09:10 CET Md. Hejbul Tawhid MUNNA wrote:
> Hi,
>
> Suddenly we have observed few VM down . then we have found some agent are
> getting down (XXX) , agents are getting UP and down randomly. Please check
> the attachment.
>
>
/////////////////////////////////////////////////////////////////////////////
> /////////////////////// /sqlalchemy/pool.py", line 788, in _checkout\n   
> fairy =
> _ConnectionRecord.checkout(pool)\n', u'  File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 532, in
> checkout\n    rec = pool._do_get()\n', u'  File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1186, in
> _do_get\n    (self.size(), self.overflow(), self._timeout),
> code="3o7r")\n', u'TimeoutError: QueuePool limit of size 5 overflow 50
> reached, connection timed out, timeout 30 (Background on this error at:
> http://sqlalche.me/e/3o7r)\n'].
> 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent Traceback (most
> recent call last):
> 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent   File
> "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 837, in
> _report_state
> 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent     True)
> 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent   File
> "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in
> report_state
> 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent     return
> method(context, 'report_state', **kwargs)
> 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent   File
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179,
> in call
> 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent
> retry=self.retry)
> 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent   File
> "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133,
> in _send
> 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent     retry=retry)
> 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent   File
> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 645, in send
> 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent
> call_monitor_timeout, retry=retry)
> 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent   File
> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 636, in _send
> 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent     raise result
> 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent RemoteError:
> Remote error: TimeoutError QueuePool limit of size 5 overflow 50 reached,
> connection timed out, timeout 30 (Background on this error at:
> http://sqlalche.me/e/3o7r)
> 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent [u'Traceback
> (most recent call last):\n', u'  File
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163,
> in _process_incoming\n    res = self.dispatcher.dispatch(message)\n', u'
>  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 265, in dispatch\n    return self._do_dispatch(endpoint, method, ctxt,
> args)\n', u'  File
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line
> 194, in _do_dispatch\n
>
>
/////////////////////////////////////////////////////////////////////////////
> ////
>
> Is there anything related with the following default configuration.
>
> /etc/neutron/neutron.conf
> #max_pool_size = 5
> #max_overflow = 50

Yes. You probably have busy environment and You need to increase those values
to have more connections from the neutron server to the database.

>
> regards,
> Munna



--
Slawek Kaplonski
Principal Software Engineer
Red Hat