On Mon, 22 Mar 2021 at 13:22, Adam Tomas <bkslash@poczta.onet.pl> wrote:
Hi Mark, this happens without any noticeable load on Openstack. Where should I put [oslo_messaging_rabbitmq] heartbeat_timeout_threshold in kolla-ansible? I can’t find any oslo.config file… Should it be in .conf file of every service? Best regards Adam
Yes, any service in which you are seeing heartbeat timeouts.
Wiadomość napisana przez Mark Goddard <mark@stackhpc.com> w dniu 22.03.2021, o godz. 10:49:
On Thu, 18 Mar 2021 at 16:05, Adam Tomas <bkslash@poczta.onet.pl> wrote:
Hi, I have a problem with rabbitmq heartbeats timeout (described also here: https://bugzilla.redhat.com/show_bug.cgi?id=1711794). Although I have threads=1 the problem still persists, generating a lot of messages in logs:
2021-03-18 15:17:17.482 [error] <0.122.51> closing AMQP connection <0.122.51> (x.x.x.100:60456 -> x.x.x.100:5672 - mod_wsgi:699:9a813fcb-c29f-4886-82bc-00bf478b6b64): missed heartbeats from client, timeout: 60s 2021-03-18 15:17:17.484 [info] <0.846.51> Closing all channels from connection '<<x.x.x.100:60456 -> x.x.x.100:5672">>' because it has been closed 2021-03-18 15:18:15.918 [error] <0.150.51> closing AMQP connection <0.150.51> (x.x.x.111:41934 -> x.x.x.100:5672 - mod_wsgi:697:b608c7b8-9644-434e-93af-c00222c0a700): missed heartbeats from client, timeout: 60s 2021-03-18 15:18:15.920 [error] <0.153.51> closing AMQP connection <0.153.51> (x.x.x.111:41936 -> x.x.x.100:5672 - mod_wsgi:697:77348197-148b-41a6-928f-c5eddfab57c9): missed heartbeats from client, timeout: 60s 2021-03-18 15:18:15.920 [info] <0.1527.51> Closing all channels from connection '<<x.x.x.111:41934 -> x.x.x.100:5672">>' because it has been closed 2021-03-18 15:18:15.922 [info] <0.1531.51> Closing all channels from connection '<<x.x.x.111:41936 -> x.x.x.100:5672">>' because it has been closed 2021-03-18 15:20:16.080 [info] <0.2196.51> accepting AMQP connection <0.2196.51> (x.x.x.111:34826 -> x.x.x.100:5672) 2021-03-18 15:20:16.080 [info] <0.2199.51> accepting AMQP connection <0.2199.51> (x.x.x.111:34828 -> x.x.x.100:5672)
I’ve set heartbeat = 600 in rabbitmq.conf and still get disconnections after 60s timeout… How to set proper timeout to avoid disconnections?
Hi Adam,
I have seen similar messages in the past, but haven't really looked into it. It seems to happen during some intensive processes like encrypted cinder volume creation.
Have you tried configuring oslo.messaging? For example, [oslo_messaging_rabbitmq] heartbeat_timeout_threshold.
https://docs.openstack.org/oslo.messaging/latest/configuration/opts.html
Mark
Best regards Adam