<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Dominic,<div class="">according to documentation:</div><div class=""><br class=""></div><div class=""><table class="lt-4-cols lt-7-rows" style="box-sizing: border-box; border-spacing: 0px; width: auto; max-width: 100%; margin-bottom: 1.25rem; font-size: 0.875rem; border-top-width: 0.0625rem; border-top-style: solid; border-top-color: rgb(209, 209, 209); border-bottom-width: 0.0625rem; border-bottom-style: solid; border-bottom-color: rgb(209, 209, 209); caret-color: rgb(37, 37, 37); color: rgb(37, 37, 37); font-family: "Red Hat Text", RedHatText, "Helvetica Neue", Arial, sans-serif;"><tbody style="box-sizing: border-box;" class=""><tr style="box-sizing: border-box; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px;" class=""><td align="left" valign="top" headers="idm139846231867808" style="box-sizing: border-box; padding: 0.5625rem 1.25rem; background-image: none; word-wrap: normal; line-height: 1.42857; vertical-align: top; border-width: 0.0625rem 0px 0px 0.0625rem; border-top-style: solid; border-top-color: rgb(237, 237, 237); border-left-style: solid; border-left-color: rgb(209, 209, 209);" class=""><p style="box-sizing: border-box; margin: 0px 0px 1.8em; line-height: 1.5em;" class=""><code class="literal" style="box-sizing: border-box; font-family: Consolas, Monaco, "Andale Mono", monospace; font-size: 12.600000381469727px; background-color: rgb(245, 245, 245); border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; padding: 0.125rem 0.25rem;">heartbeat_rate = 2</code></p></td><td align="left" valign="top" headers="idm139846231866704" style="box-sizing: border-box; padding: 0.5625rem 1.25rem; background-image: none; word-wrap: normal; line-height: 1.42857; vertical-align: top; border-width: 0.0625rem 0px 0px 0.0625rem; border-top-style: solid; border-top-color: rgb(237, 237, 237); border-left-style: solid; border-left-color: rgb(209, 209, 209);" class=""><p style="box-sizing: border-box; margin: 0px 0px 1.8em; line-height: 1.5em;" class="">integer value</p></td><td align="left" valign="top" headers="idm139846231865616" style="box-sizing: border-box; padding: 0.5625rem 1.25rem; background-image: none; word-wrap: normal; line-height: 1.42857; vertical-align: top; border-width: 0.0625rem 0.0625rem 0px; border-top-style: solid; border-top-color: rgb(237, 237, 237); border-right-style: solid; border-right-color: rgb(209, 209, 209); border-left-style: solid; border-left-color: rgb(209, 209, 209);" class=""><p style="box-sizing: border-box; margin: 0px 0px 1.8em; line-height: 1.5em;" class="">How often times during the heartbeat_timeout_threshold we check the heartbeat.</p></td></tr><tr style="box-sizing: border-box; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px;" class=""><td align="left" valign="top" headers="idm139846231867808" style="box-sizing: border-box; padding: 0.5625rem 1.25rem; background-image: none; word-wrap: normal; line-height: 1.42857; vertical-align: top; border-width: 0.0625rem 0px 0px 0.0625rem; border-top-style: solid; border-top-color: rgb(237, 237, 237); border-left-style: solid; border-left-color: rgb(209, 209, 209);" class=""><p style="box-sizing: border-box; margin: 0px 0px 1.8em; line-height: 1.5em;" class=""><code class="literal" style="box-sizing: border-box; font-family: Consolas, Monaco, "Andale Mono", monospace; font-size: 12.600000381469727px; background-color: rgb(245, 245, 245); border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; padding: 0.125rem 0.25rem;">heartbeat_timeout_threshold = 60</code></p></td><td align="left" valign="top" headers="idm139846231866704" style="box-sizing: border-box; padding: 0.5625rem 1.25rem; background-image: none; word-wrap: normal; line-height: 1.42857; vertical-align: top; border-width: 0.0625rem 0px 0px 0.0625rem; border-top-style: solid; border-top-color: rgb(237, 237, 237); border-left-style: solid; border-left-color: rgb(209, 209, 209);" class=""><p style="box-sizing: border-box; margin: 0px 0px 1.8em; line-height: 1.5em;" class="">integer value</p></td><td align="left" valign="top" headers="idm139846231865616" style="box-sizing: border-box; padding: 0.5625rem 1.25rem; background-image: none; word-wrap: normal; line-height: 1.42857; vertical-align: top; border-width: 0.0625rem 0.0625rem 0px; border-top-style: solid; border-top-color: rgb(237, 237, 237); border-right-style: solid; border-right-color: rgb(209, 209, 209); border-left-style: solid; border-left-color: rgb(209, 209, 209);" class=""><p style="box-sizing: border-box; margin: 0px 0px 1.8em; line-height: 1.5em;" class="">Number of seconds after which the Rabbit broker is considered down if heartbeat’s keep-alive fails (0 disables heartbeat).</p></td></tr></tbody></table><div class="">So to avoid disconnection from service side (nova, keystone, etc.) I’ve increased Heartbeat_timeout_treshold from 60 to 720 and set heartbeat_rate to 4, so Rabbit will be considered dead after 360s not after 60s as in default. In addition I’ve increased heartbeat to 600 from 60 in rabbitmq.conf. And again - according to documentation:</div><div class=""><br class=""></div><div class=""><span style="font-family: Raleway, sans-serif; font-size: 16px; letter-spacing: 0.1599999964237213px;" class="">The </span><span class=" code" style="color: white; background-color: rgb(120, 120, 120); font-family: monospace; font-size: medium; padding: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; word-wrap: break-word; white-space: pre-wrap; letter-spacing: 0.1599999964237213px;">heartbeat timeout</span><span style="font-family: Raleway, sans-serif; font-size: 16px; letter-spacing: 0.1599999964237213px;" class=""> value defines after what period of time the peer TCP connection should be considered unreachable (down) by RabbitMQ and client libraries.</span></div><div><br class=""></div><div>So disconnection from Rabbit’s side should be after 600s of no heartbeat</div><div><br class=""></div><div>And I still got disconnections.</div><div><br class=""></div><div>Best regards</div><div>Adam Tomas</div><div><br class=""><blockquote type="cite" class=""><div class="">Wiadomość napisana przez <a href="mailto:DHilsbos@performair.com" class="">DHilsbos@performair.com</a> w dniu 17.09.2021, o godz. 18:38:</div><br class="Apple-interchange-newline"><div class=""><div class="">Adam;<br class=""><br class="">If I'm reading this correctly; Rabbit is timing out, but you're increasing the heartbeat period of OpenStack. This would make the issue worse, wouldn't it? <br class=""><br class="">It seems to me that you would want to lower the heartbeat interval of OpenStack, and raise the timeout of Rabbit.<br class=""><br class="">That said; it looks like you're using Kola, and I know nothing about Kola.<br class=""><br class="">Thank you,<br class=""><br class="">Dominic L. Hilsbos, MBA<br class="">Vice President – Information Technology<br class="">Perform Air International Inc.<br class=""><a href="mailto:DHilsbos@PerformAir.com" class="">DHilsbos@PerformAir.com</a><br class="">www.PerformAir.com<br class=""><br class=""><br class="">From: Adam Tomas [mailto:bkslash@poczta.onet.pl] <br class="">Sent: Friday, September 17, 2021 5:01 AM<br class="">To: openstack-discuss<br class="">Subject: RabbitMQ annoying disconnections<br class=""><br class="">Hi,<br class="">after some struggling I have almost „clear” logs (clear=error free :) ) Almost…. RabbitMQ keeps disconnecting sessions and there is a huge amount of disconnect errors in all logs (see below). I found this bug description:<br class=""><br class="">https://bugzilla.redhat.com/show_bug.cgi?id=1711794<br class=""><br class="">in which we can read as follows: "this is a recoverable issue that is already handled by how oslo.messaging is designed. disconnection is not an error and should not be reported as such in the logs.”<br class=""><br class=""><br class="">but… It is reported :( And produces tons of logs.<br class=""><br class=""><br class="">I tried to modify heartbeat values - helped a little bit, but I had to increase [database] max_pool_size = 5 and that of course multiplied number of disconnection errors by 5 :( <br class=""><br class=""><br class="">[oslo_messaging_rabbit]<br class="">heartbeat_timeout_threshold = 720 heartbeat_interval = 360 heartbeat_rate = 4<br class=""><br class=""><br class=""></div></div></blockquote></div><br class=""></div></body></html>