Open Stack

Folks,

I am getting a very strange error from time to time on senlin logs and
when it hits this error i can't do anything in senlin until i restart
service.

After googling i found this bug
https://bugs.launchpad.net/oslo.messaging/+bug/1905965

It's pretty much related to my issue because I am running the latest victoria.

Dec 22 13:56:44 os-lab-infra-1-senlin-container-16f24bbe
senlin-wsgi-api[8188]: 2020-12-22 13:56:44.212 8188 ERROR
oslo.messaging._drivers.impl_rabbit [-]
[df314561-6415-4103-a1fd-14ab95182cfb] AMQP server on 10.65.6.176:5671
is unreachable: <RecoverableConnectionError: unknown error>. Trying
again in 1 seconds.: amqp.exceptions.RecoverableConnectionError:
<RecoverableConnectionError: unknown error>
Dec 22 13:56:44 os-lab-infra-1-senlin-container-16f24bbe
senlin-wsgi-api[8188]: 2020-12-22 13:56:44.220 8188 INFO
oslo.messaging._drivers.impl_rabbit [-] A recoverable
connection/channel error occurred, trying to reconnect: Server
unexpectedly closed connection
Dec 22 13:56:44 os-lab-infra-1-senlin-container-16f24bbe
senlin-conductor[8250]: 2020-12-22 13:56:44.461 8250 ERROR
oslo_messaging.rpc.server [req-3c89475b-89fc-404b-8537-df7a587261d9
462618bed32745d2a9166bcc33fc117e f1502c79c70f4651be8ffc7b844b584f - -
-] MessageUndeliverable error, source exception: Basic.return: (312)
NO_ROUTE, routing_key: reply_54d93c43fe894ed18ce8092f4497306b,
exchange: : : oslo_messaging.exceptions.MessageUndeliverable

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
Traceback (most recent call last):

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/rpc/server.py",
line 184, in _process_incoming

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    message.reply(res)

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 149, in reply

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    self._send_reply(conn, reply, failure)

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 125, in _send_reply

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg))

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py",
line 1320, in direct_send

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    self._ensure_publishing(self._publish_and_raises_on_missing_exchange,

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py",
line 1202, in _ensure_publishing

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    self.ensure(method, retry=retry, error_callback=_error_callback)

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py",
line 826, in ensure

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    ret, channel = autoretry_method()

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/kombu/connection.py",
line 525, in _ensured

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    return fun(*args, **kwargs)

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/kombu/connection.py",
line 601, in __call__

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    return fun(*args, channel=channels[0], **kwargs), channels[0]

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py",
line 815, in execute_method

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    method()

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py",
line 1294, in _publish_and_raises_on_missing_exchange

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    self._publish(exchange, msg, routing_key=routing_key,

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py",
line 1238, in _publish

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    self._producer.publish(

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/kombu/messaging.py",
line 175, in publish

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    return _publish(

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/kombu/messaging.py",
line 197, in _publish

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    return channel.basic_publish(

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/channel.py",
line 1782, in basic_publish_confirm

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    self.wait([spec.Basic.Ack, spec.Basic.Nack],
callback=confirm_handler)

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/abstract_channel.py",
line 86, in wait

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    self.connection.drain_events(timeout=timeout)

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/connection.py",
line 514, in drain_events

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    while not self.blocking_read(timeout):

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/connection.py",
line 520, in blocking_read

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    return self.on_inbound_frame(frame)

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/method_framing.py",
line 77, in on_frame

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    callback(channel, msg.frame_method, msg.frame_args, msg)

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/connection.py",
line 526, in on_inbound_method

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    return self.channels[channel_id].dispatch_method(

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/abstract_channel.py",
line 143, in dispatch_method

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    listener(*args)

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/channel.py",
line 2006, in _on_basic_return

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    callback(exc, exchange, routing_key, message)

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
  File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py",
line 852, in on_return

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
    raise exceptions.MessageUndeliverable(exception, exchange,
routing_key,

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
oslo_messaging.exceptions.MessageUndeliverable

          2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server
Dec 22 13:56:45 os-lab-infra-1-senlin-container-16f24bbe
senlin-wsgi-api[8188]: 2020-12-22 13:56:45.248 8188 INFO
oslo.messaging._drivers.impl_rabbit [-]
[df314561-6415-4103-a1fd-14ab95182cfb] Reconnected to AMQP server on
10.65.6.176:5671 via [amqp] client with port 44246.
Dec 22 13:56:53 os-lab-infra-1-senlin-container-16f24bbe
senlin-health-manager[8218]: 2020-12-22 13:56:53.954 8218 INFO
senlin.engine.health_manager [req-c0118d24-9f32-47c8-926e-eb4b60f922cd
462618bed32745d2a9166bcc33fc117e f1502c79c70f4651be8ffc7b844b584f - -
-] Health check passed for all nodes in cluster
f51b3154-fb8e-43f2-b68b-03b091c1e0bf.
Dec 22 13:57:23 os-lab-infra-1-senlin-container-16f24bbe
senlin-health-manager[8218]: 2020-12-22 13:57:23.962 8218 INFO
senlin.engine.health_manager [req-472f4403-fd66-422a-8a84-a0b0f51d94cc
462618bed32745d2a9166bcc33fc117e f1502c79c70f4651be8ffc7b844b584f - -
-] Health check passed for all nodes in cluster
f51b3154-fb8e-43f2-b68b-03b091c1e0bf.
Dec 22 13:57:23 os-lab-infra-1-senlin-container-16f24bbe
senlin-health-manager[8218]: 2020-12-22 13:57:23.963 8218 INFO
senlin.engine.health_manager [req-a5ff1fe3-35e8-443b-b91a-aa1d78656569
462618bed32745d2a9166bcc33fc117e f1502c79c70f4651be8ffc7b844b584f - -
-] Health check passed for all nodes in cluster
f51b3154-fb8e-43f2-b68b-03b091c1e0bf.
Dec 22 13:57:44 os-lab-infra-1-senlin-container-16f24bbe uwsgi[8188]:
Tue Dec 22 13:57:44 2020 -
uwsgi_response_writev_headers_and_body_do(): Connection reset by peer
[core/writer.c line 306] during GET /v1/clusters?global_project=False
(10.65.6.17)
Dec 22 13:57:44 os-lab-infra-1-senlin-container-16f24bbe
senlin-wsgi-api[8188]: 2020-12-22 13:57:44.448 8188 CRITICAL
senlin-api [req-3c89475b-89fc-404b-8537-df7a587261d9
462618bed32745d2a9166bcc33fc117e f1502c79c70f4651be8ffc7b844b584f - -
-] Unhandled error: OSError: write error

         2020-12-22 13:57:44.448 8188 ERROR senlin-api OSError: write
error

         2020-12-22 13:57:44.448 8188 ERROR senlin-api
Dec 22 13:57:53 os-lab-infra-1-senlin-container-16f24bbe
senlin-health-manager[8218]: 2020-12-22 13:57:53.955 8218 INFO
senlin.engine.health_manager [req-8a578e8e-438d-45c1-b81d-c6f3aed5fc81
462618bed32745d2a9166bcc33fc117e f1502c79c70f4651be8ffc7b844b584f - -
-] Health check passed for all nodes in cluster
f51b3154-fb8e-43f2-b68b-03b091c1e0bf.


As per advice i have tired to add following snippet to fix this issue

[DEFAULT]
transport_url =
rabbit://senlin:94d7aecb853145779db8f1dcb@10.65.6.176:5671//senlin?ssl=1

[oslo_messaging_rabbit]
ssl = True
direct_mandatory_flag = False

But i got error when trying to set direct_mandatory_flag option

ERROR oslo_service.service [-] Error starting t
hread.: oslo_config.cfg.ConfigFileValueError: Value for option
direct_mandatory_flag from LocationInfo(location=<Locations.user: (4,
True)>, detail='/etc/senlin/senlin.conf') is not
 valid: invalid literal for int() with base 10: 'False'

Any idea what is going on here?