`heartbeat_rate = 2`	integer value	How often times during the heartbeat_timeout_threshold we check the heartbeat.
`heartbeat_timeout_threshold = 60`	integer value	Number of seconds after which the Rabbit broker is considered down if heartbeat’s keep-alive fails (0 disables heartbeat).

So to avoid disconnection from service side (nova, keystone, etc.) I’ve increased Heartbeat_timeout_treshold from 60 to 720 and set heartbeat_rate to 4, so Rabbit will be considered dead after 360s not after 60s as in default. In addition I’ve increased heartbeat to 600 from 60 in rabbitmq.conf. And again - according to documentation:

The heartbeat timeout value defines after what period of time the peer TCP connection should be considered unreachable (down) by RabbitMQ and client libraries.

So disconnection from Rabbit’s side should be after 600s of no heartbeat

And I still got disconnections.

Best regards

Adam Tomas

Wiadomość napisana przez DHilsbos@performair.com w dniu 17.09.2021, o godz. 18:38:

Adam;

If I'm reading this correctly; Rabbit is timing out, but you're increasing the heartbeat period of OpenStack. This would make the issue worse, wouldn't it?

It seems to me that you would want to lower the heartbeat interval of OpenStack, and raise the timeout of Rabbit.

That said; it looks like you're using Kola, and I know nothing about Kola.

Thank you,

Dominic L. Hilsbos, MBA
Vice President – Information Technology
Perform Air International Inc.
DHilsbos@PerformAir.com
www.PerformAir.com

From: Adam Tomas [mailto:bkslash@poczta.onet.pl]
Sent: Friday, September 17, 2021 5:01 AM
To: openstack-discuss
Subject: RabbitMQ annoying disconnections

Hi,
after some struggling I have almost „clear” logs (clear=error free :) ) Almost…. RabbitMQ keeps disconnecting sessions and there is a huge amount of disconnect errors in all logs (see below). I found this bug description:

https://bugzilla.redhat.com/show_bug.cgi?id=1711794

in which we can read as follows: "this is a recoverable issue that is already handled by how oslo.messaging is designed. disconnection is not an error and should not be reported as such in the logs.”

but… It is reported :( And produces tons of logs.

I tried to modify heartbeat values - helped a little bit, but I had to increase [database] max_pool_size = 5 and that of course multiplied number of disconnection errors by 5 :(

[oslo_messaging_rabbit]
heartbeat_timeout_threshold = 720 heartbeat_interval = 360 heartbeat_rate = 4