Open Stack

Hi,

I am not an db expert, but openstack tends to open a LOT of connection to the db. So one of the thing to monitor is the number of connection you have/allow on the db side.

Also, raising the number of RPC (and report states) workers will solve your issue.
The good number is not easy to calculate, and depends on each deployment.

A good approach is to the try/improve loop.

Cheers,
Arnaud.


Le 25 janvier 2022 09:52:26 GMT+01:00, "Md. Hejbul Tawhid MUNNA" <munnaeebd at gmail.com> a écrit :
>Hello  Arnaud,
>
>Thank you for your valuable reply.
>
>we did not modify default config of RPC worker .
>
>/etc/neutron/neutron.conf
>
># Number of separate API worker processes for service. If not specified, the
># default is equal to the number of CPUs available for best performance.
># (integer value)
>#api_workers = <None>
>
># Number of RPC worker processes for service. (integer value)
>#rpc_workers = 1
>
># Number of RPC worker processes dedicated to state reports queue. (integer
># value)
>#rpc_state_report_workers = 1
>
>how to check load on database. RAM/CPU/Disk-IO utilization is low on the
>database server.
>
>Please guide us further
>
>Regards,
>Munna
>
>On Mon, Jan 24, 2022 at 6:29 PM Arnaud <arnaud.morin at gmail.com> wrote:
>
>> Hi,
>>
>> I would also consider checking the number of RPC workers you have in
>> neutron.conf, this is maybe a better option to increase this before the
>> comnection pool params.
>>
>> Also, check your database, is it under load?
>> Updating agent state should not be long.
>>
>> Cheers,
>> Arnaud
>>
>>
>>
>> Le 24 janvier 2022 10:42:00 GMT+01:00, "Md. Hejbul Tawhid MUNNA" <
>> munnaeebd at gmail.com> a écrit :
>>>
>>> Hi,
>>>
>>> Currently we have running 500+VM and total network is 383 including
>>> HA-network.
>>>
>>> Can you advice the appropriate value and is there any chance of service
>>> impact?
>>>
>>> Should we change the configuration in the neutron.conf on controller node?
>>>
>>> Regards,
>>> Munna
>>>
>>>
>>>
>>> On Mon, Jan 24, 2022 at 2:47 PM Slawek Kaplonski <skaplons at redhat.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> On poniedziałek, 24 stycznia 2022 09:09:10 CET Md. Hejbul Tawhid MUNNA
>>>> wrote:
>>>> > Hi,
>>>> >
>>>> > Suddenly we have observed few VM down . then we have found some agent
>>>> are
>>>> > getting down (XXX) , agents are getting UP and down randomly. Please
>>>> check
>>>> > the attachment.
>>>> >
>>>> >
>>>>
>>>> /////////////////////////////////////////////////////////////////////////////
>>>> > /////////////////////// /sqlalchemy/pool.py", line 788, in
>>>> _checkout\n
>>>> > fairy =
>>>> > _ConnectionRecord.checkout(pool)\n', u'  File
>>>> > "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 532, in
>>>> > checkout\n    rec = pool._do_get()\n', u'  File
>>>> > "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1186, in
>>>> > _do_get\n    (self.size(), self.overflow(), self._timeout),
>>>> > code="3o7r")\n', u'TimeoutError: QueuePool limit of size 5 overflow 50
>>>> > reached, connection timed out, timeout 30 (Background on this error at:
>>>> > http://sqlalche.me/e/3o7r)\n'].
>>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent Traceback
>>>> (most
>>>> > recent call last):
>>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent   File
>>>> > "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line
>>>> 837, in
>>>> > _report_state
>>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent     True)
>>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent   File
>>>> > "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in
>>>> > report_state
>>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent     return
>>>> > method(context, 'report_state', **kwargs)
>>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent   File
>>>> > "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line
>>>> 179,
>>>> > in call
>>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent
>>>> > retry=self.retry)
>>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent   File
>>>> > "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line
>>>> 133,
>>>> > in _send
>>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent
>>>>  retry=retry)
>>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent   File
>>>> >
>>>> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
>>>> > line 645, in send
>>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent
>>>> > call_monitor_timeout, retry=retry)
>>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent   File
>>>> >
>>>> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
>>>> > line 636, in _send
>>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent     raise
>>>> result
>>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent
>>>> RemoteError:
>>>> > Remote error: TimeoutError QueuePool limit of size 5 overflow 50
>>>> reached,
>>>> > connection timed out, timeout 30 (Background on this error at:
>>>> > http://sqlalche.me/e/3o7r)
>>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent
>>>> [u'Traceback
>>>> > (most recent call last):\n', u'  File
>>>> > "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line
>>>> 163,
>>>> > in _process_incoming\n    res = self.dispatcher.dispatch(message)\n',
>>>> u'
>>>> >  File
>>>> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
>>>> > line 265, in dispatch\n    return self._do_dispatch(endpoint, method,
>>>> ctxt,
>>>> > args)\n', u'  File
>>>> > "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
>>>> line
>>>> > 194, in _do_dispatch\n
>>>> >
>>>> >
>>>>
>>>> /////////////////////////////////////////////////////////////////////////////
>>>> > ////
>>>> >
>>>> > Is there anything related with the following default configuration.
>>>> >
>>>> > /etc/neutron/neutron.conf
>>>> > #max_pool_size = 5
>>>> > #max_overflow = 50
>>>>
>>>> Yes. You probably have busy environment and You need to increase those
>>>> values
>>>> to have more connections from the neutron server to the database.
>>>>
>>>> >
>>>> > regards,
>>>> > Munna
>>>>
>>>>
>>>>
>>>> --
>>>> Slawek Kaplonski
>>>> Principal Software Engineer
>>>> Red Hat
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220125/cee9a3b9/attachment.htm>