Open Stack

On Mon, 22 Mar 2021 at 13:22, Adam Tomas <bkslash at poczta.onet.pl> wrote:
>
> Hi Mark,
> this happens without any noticeable load on Openstack. Where should I put [oslo_messaging_rabbitmq] heartbeat_timeout_threshold in kolla-ansible? I can’t find any oslo.config file… Should it be in  .conf file of every service?
> Best regards
> Adam

Yes, any service in which you are seeing heartbeat timeouts.
>
>
> Wiadomość napisana przez Mark Goddard <mark at stackhpc.com> w dniu 22.03.2021, o godz. 10:49:
>
> On Thu, 18 Mar 2021 at 16:05, Adam Tomas <bkslash at poczta.onet.pl> wrote:
>
>
> Hi,
> I have a problem with rabbitmq heartbeats timeout (described also here: https://bugzilla.redhat.com/show_bug.cgi?id=1711794). Although I have threads=1 the problem still persists, generating a lot of messages in logs:
>
> 2021-03-18 15:17:17.482 [error] <0.122.51> closing AMQP connection <0.122.51> (x.x.x.100:60456 -> x.x.x.100:5672 - mod_wsgi:699:9a813fcb-c29f-4886-82bc-00bf478b6b64):
> missed heartbeats from client, timeout: 60s
> 2021-03-18 15:17:17.484 [info] <0.846.51> Closing all channels from connection '<<x.x.x.100:60456 -> x.x.x.100:5672">>' because it has been closed
> 2021-03-18 15:18:15.918 [error] <0.150.51> closing AMQP connection <0.150.51> (x.x.x.111:41934 -> x.x.x.100:5672 - mod_wsgi:697:b608c7b8-9644-434e-93af-c00222c0a700):
> missed heartbeats from client, timeout: 60s
> 2021-03-18 15:18:15.920 [error] <0.153.51> closing AMQP connection <0.153.51> (x.x.x.111:41936 -> x.x.x.100:5672 - mod_wsgi:697:77348197-148b-41a6-928f-c5eddfab57c9):
> missed heartbeats from client, timeout: 60s
> 2021-03-18 15:18:15.920 [info] <0.1527.51> Closing all channels from connection '<<x.x.x.111:41934 -> x.x.x.100:5672">>' because it has been closed
> 2021-03-18 15:18:15.922 [info] <0.1531.51> Closing all channels from connection '<<x.x.x.111:41936 -> x.x.x.100:5672">>' because it has been closed
> 2021-03-18 15:20:16.080 [info] <0.2196.51> accepting AMQP connection <0.2196.51> (x.x.x.111:34826 -> x.x.x.100:5672)
> 2021-03-18 15:20:16.080 [info] <0.2199.51> accepting AMQP connection <0.2199.51> (x.x.x.111:34828 -> x.x.x.100:5672)
>
> I’ve set heartbeat = 600 in rabbitmq.conf and still get disconnections after 60s timeout… How to set proper timeout to avoid disconnections?
>
>
> Hi Adam,
>
> I have seen similar messages in the past, but haven't really looked
> into it. It seems to happen during some intensive processes like
> encrypted cinder volume creation.
>
> Have you tried configuring oslo.messaging? For example,
> [oslo_messaging_rabbitmq] heartbeat_timeout_threshold.
>
> https://docs.openstack.org/oslo.messaging/latest/configuration/opts.html
>
> Mark
>
>
> Best regards
> Adam
>
>