Open Stack

Hello  Arnaud,

Thank you for your valuable reply.

we did not modify default config of RPC worker .

/etc/neutron/neutron.conf

# Number of separate API worker processes for service. If not specified, the
# default is equal to the number of CPUs available for best performance.
# (integer value)
#api_workers = <None>

# Number of RPC worker processes for service. (integer value)
#rpc_workers = 1

# Number of RPC worker processes dedicated to state reports queue. (integer
# value)
#rpc_state_report_workers = 1

how to check load on database. RAM/CPU/Disk-IO utilization is low on the
database server.

Please guide us further

Regards,
Munna

On Mon, Jan 24, 2022 at 6:29 PM Arnaud <arnaud.morin at gmail.com> wrote:

> Hi,
>
> I would also consider checking the number of RPC workers you have in
> neutron.conf, this is maybe a better option to increase this before the
> comnection pool params.
>
> Also, check your database, is it under load?
> Updating agent state should not be long.
>
> Cheers,
> Arnaud
>
>
>
> Le 24 janvier 2022 10:42:00 GMT+01:00, "Md. Hejbul Tawhid MUNNA" <
> munnaeebd at gmail.com> a écrit :
>>
>> Hi,
>>
>> Currently we have running 500+VM and total network is 383 including
>> HA-network.
>>
>> Can you advice the appropriate value and is there any chance of service
>> impact?
>>
>> Should we change the configuration in the neutron.conf on controller node?
>>
>> Regards,
>> Munna
>>
>>
>>
>> On Mon, Jan 24, 2022 at 2:47 PM Slawek Kaplonski <skaplons at redhat.com>
>> wrote:
>>
>>> Hi,
>>>
>>> On poniedziałek, 24 stycznia 2022 09:09:10 CET Md. Hejbul Tawhid MUNNA
>>> wrote:
>>> > Hi,
>>> >
>>> > Suddenly we have observed few VM down . then we have found some agent
>>> are
>>> > getting down (XXX) , agents are getting UP and down randomly. Please
>>> check
>>> > the attachment.
>>> >
>>> >
>>>
>>> /////////////////////////////////////////////////////////////////////////////
>>> > /////////////////////// /sqlalchemy/pool.py", line 788, in
>>> _checkout\n
>>> > fairy =
>>> > _ConnectionRecord.checkout(pool)\n', u'  File
>>> > "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 532, in
>>> > checkout\n    rec = pool._do_get()\n', u'  File
>>> > "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1186, in
>>> > _do_get\n    (self.size(), self.overflow(), self._timeout),
>>> > code="3o7r")\n', u'TimeoutError: QueuePool limit of size 5 overflow 50
>>> > reached, connection timed out, timeout 30 (Background on this error at:
>>> > http://sqlalche.me/e/3o7r)\n'].
>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent Traceback
>>> (most
>>> > recent call last):
>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent   File
>>> > "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line
>>> 837, in
>>> > _report_state
>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent     True)
>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent   File
>>> > "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in
>>> > report_state
>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent     return
>>> > method(context, 'report_state', **kwargs)
>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent   File
>>> > "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line
>>> 179,
>>> > in call
>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent
>>> > retry=self.retry)
>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent   File
>>> > "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line
>>> 133,
>>> > in _send
>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent
>>>  retry=retry)
>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent   File
>>> >
>>> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
>>> > line 645, in send
>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent
>>> > call_monitor_timeout, retry=retry)
>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent   File
>>> >
>>> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
>>> > line 636, in _send
>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent     raise
>>> result
>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent
>>> RemoteError:
>>> > Remote error: TimeoutError QueuePool limit of size 5 overflow 50
>>> reached,
>>> > connection timed out, timeout 30 (Background on this error at:
>>> > http://sqlalche.me/e/3o7r)
>>> > 2022-01-24 01:05:39.592 302841 ERROR neutron.agent.l3.agent
>>> [u'Traceback
>>> > (most recent call last):\n', u'  File
>>> > "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line
>>> 163,
>>> > in _process_incoming\n    res = self.dispatcher.dispatch(message)\n',
>>> u'
>>> >  File
>>> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
>>> > line 265, in dispatch\n    return self._do_dispatch(endpoint, method,
>>> ctxt,
>>> > args)\n', u'  File
>>> > "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
>>> line
>>> > 194, in _do_dispatch\n
>>> >
>>> >
>>>
>>> /////////////////////////////////////////////////////////////////////////////
>>> > ////
>>> >
>>> > Is there anything related with the following default configuration.
>>> >
>>> > /etc/neutron/neutron.conf
>>> > #max_pool_size = 5
>>> > #max_overflow = 50
>>>
>>> Yes. You probably have busy environment and You need to increase those
>>> values
>>> to have more connections from the neutron server to the database.
>>>
>>> >
>>> > regards,
>>> > Munna
>>>
>>>
>>>
>>> --
>>> Slawek Kaplonski
>>> Principal Software Engineer
>>> Red Hat
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220125/3a6cb6ca/attachment-0001.htm>