Folks, I am getting a very strange error from time to time on senlin logs and when it hits this error i can't do anything in senlin until i restart service. After googling i found this bug https://bugs.launchpad.net/oslo.messaging/+bug/1905965 It's pretty much related to my issue because I am running the latest victoria. Dec 22 13:56:44 os-lab-infra-1-senlin-container-16f24bbe senlin-wsgi-api[8188]: 2020-12-22 13:56:44.212 8188 ERROR oslo.messaging._drivers.impl_rabbit [-] [df314561-6415-4103-a1fd-14ab95182cfb] AMQP server on 10.65.6.176:5671 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error> Dec 22 13:56:44 os-lab-infra-1-senlin-container-16f24bbe senlin-wsgi-api[8188]: 2020-12-22 13:56:44.220 8188 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Dec 22 13:56:44 os-lab-infra-1-senlin-container-16f24bbe senlin-conductor[8250]: 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server [req-3c89475b-89fc-404b-8537-df7a587261d9 462618bed32745d2a9166bcc33fc117e f1502c79c70f4651be8ffc7b844b584f - - -] MessageUndeliverable error, source exception: Basic.return: (312) NO_ROUTE, routing_key: reply_54d93c43fe894ed18ce8092f4497306b, exchange: : : oslo_messaging.exceptions.MessageUndeliverable 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 184, in _process_incoming 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server message.reply(res) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 149, in reply 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server self._send_reply(conn, reply, failure) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 125, in _send_reply 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg)) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1320, in direct_send 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server self._ensure_publishing(self._publish_and_raises_on_missing_exchange, 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1202, in _ensure_publishing 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server self.ensure(method, retry=retry, error_callback=_error_callback) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 826, in ensure 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server ret, channel = autoretry_method() 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/kombu/connection.py", line 525, in _ensured 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server return fun(*args, **kwargs) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/kombu/connection.py", line 601, in __call__ 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server return fun(*args, channel=channels[0], **kwargs), channels[0] 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 815, in execute_method 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server method() 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1294, in _publish_and_raises_on_missing_exchange 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server self._publish(exchange, msg, routing_key=routing_key, 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1238, in _publish 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server self._producer.publish( 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/kombu/messaging.py", line 175, in publish 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server return _publish( 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/kombu/messaging.py", line 197, in _publish 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server return channel.basic_publish( 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/channel.py", line 1782, in basic_publish_confirm 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server self.wait([spec.Basic.Ack, spec.Basic.Nack], callback=confirm_handler) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/abstract_channel.py", line 86, in wait 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server self.connection.drain_events(timeout=timeout) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/connection.py", line 514, in drain_events 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server while not self.blocking_read(timeout): 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/connection.py", line 520, in blocking_read 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server return self.on_inbound_frame(frame) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/method_framing.py", line 77, in on_frame 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server callback(channel, msg.frame_method, msg.frame_args, msg) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/connection.py", line 526, in on_inbound_method 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server return self.channels[channel_id].dispatch_method( 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/abstract_channel.py", line 143, in dispatch_method 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server listener(*args) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/channel.py", line 2006, in _on_basic_return 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server callback(exc, exchange, routing_key, message) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 852, in on_return 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server raise exceptions.MessageUndeliverable(exception, exchange, routing_key, 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server oslo_messaging.exceptions.MessageUndeliverable 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server Dec 22 13:56:45 os-lab-infra-1-senlin-container-16f24bbe senlin-wsgi-api[8188]: 2020-12-22 13:56:45.248 8188 INFO oslo.messaging._drivers.impl_rabbit [-] [df314561-6415-4103-a1fd-14ab95182cfb] Reconnected to AMQP server on 10.65.6.176:5671 via [amqp] client with port 44246. Dec 22 13:56:53 os-lab-infra-1-senlin-container-16f24bbe senlin-health-manager[8218]: 2020-12-22 13:56:53.954 8218 INFO senlin.engine.health_manager [req-c0118d24-9f32-47c8-926e-eb4b60f922cd 462618bed32745d2a9166bcc33fc117e f1502c79c70f4651be8ffc7b844b584f - - -] Health check passed for all nodes in cluster f51b3154-fb8e-43f2-b68b-03b091c1e0bf. Dec 22 13:57:23 os-lab-infra-1-senlin-container-16f24bbe senlin-health-manager[8218]: 2020-12-22 13:57:23.962 8218 INFO senlin.engine.health_manager [req-472f4403-fd66-422a-8a84-a0b0f51d94cc 462618bed32745d2a9166bcc33fc117e f1502c79c70f4651be8ffc7b844b584f - - -] Health check passed for all nodes in cluster f51b3154-fb8e-43f2-b68b-03b091c1e0bf. Dec 22 13:57:23 os-lab-infra-1-senlin-container-16f24bbe senlin-health-manager[8218]: 2020-12-22 13:57:23.963 8218 INFO senlin.engine.health_manager [req-a5ff1fe3-35e8-443b-b91a-aa1d78656569 462618bed32745d2a9166bcc33fc117e f1502c79c70f4651be8ffc7b844b584f - - -] Health check passed for all nodes in cluster f51b3154-fb8e-43f2-b68b-03b091c1e0bf. Dec 22 13:57:44 os-lab-infra-1-senlin-container-16f24bbe uwsgi[8188]: Tue Dec 22 13:57:44 2020 - uwsgi_response_writev_headers_and_body_do(): Connection reset by peer [core/writer.c line 306] during GET /v1/clusters?global_project=False (10.65.6.17) Dec 22 13:57:44 os-lab-infra-1-senlin-container-16f24bbe senlin-wsgi-api[8188]: 2020-12-22 13:57:44.448 8188 CRITICAL senlin-api [req-3c89475b-89fc-404b-8537-df7a587261d9 462618bed32745d2a9166bcc33fc117e f1502c79c70f4651be8ffc7b844b584f - - -] Unhandled error: OSError: write error 2020-12-22 13:57:44.448 8188 ERROR senlin-api OSError: write error 2020-12-22 13:57:44.448 8188 ERROR senlin-api Dec 22 13:57:53 os-lab-infra-1-senlin-container-16f24bbe senlin-health-manager[8218]: 2020-12-22 13:57:53.955 8218 INFO senlin.engine.health_manager [req-8a578e8e-438d-45c1-b81d-c6f3aed5fc81 462618bed32745d2a9166bcc33fc117e f1502c79c70f4651be8ffc7b844b584f - - -] Health check passed for all nodes in cluster f51b3154-fb8e-43f2-b68b-03b091c1e0bf. As per advice i have tired to add following snippet to fix this issue [DEFAULT] transport_url = rabbit://senlin:94d7aecb853145779db8f1dcb@10.65.6.176:5671//senlin?ssl=1 [oslo_messaging_rabbit] ssl = True direct_mandatory_flag = False But i got error when trying to set direct_mandatory_flag option ERROR oslo_service.service [-] Error starting t hread.: oslo_config.cfg.ConfigFileValueError: Value for option direct_mandatory_flag from LocationInfo(location=<Locations.user: (4, True)>, detail='/etc/senlin/senlin.conf') is not valid: invalid literal for int() with base 10: 'False' Any idea what is going on here?