Hi Oliver, Looking at the output - the problem is with connection to etcd coordination backend. Best regards, Michal
On 31 May 2023, at 07:07, Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi,
after replacing my control nodes with new nodes (all bare-metal) somehow the cinder containers are no longer starting.
I checked the logs on one of the control nodes and I see this in the api-eror.log:
2023-05-30 21:31:36.350636 Timeout when reading response headers from daemon process 'cinder-api': /var/www/cgi-bin/cinder/cinder-wsgi 2023-05-30 21:31:37.827101 mod_wsgi (pid=22): Failed to exec Python script file '/var/www/cgi-bin/cinder/cinder-wsgi'. 2023-05-30 21:31:37.827168 mod_wsgi (pid=22): Exception occurred processing WSGI script '/var/www/cgi-bin/cinder/cinder-wsgi'. 2023-05-30 21:31:37.828005 Traceback (most recent call last): 2023-05-30 21:31:37.828046 File "/var/www/cgi-bin/cinder/cinder-wsgi", line 52, in <module> 2023-05-30 21:31:37.828053 application = initialize_application() 2023-05-30 21:31:37.828058 File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/wsgi/wsgi.py", line 44, in initialize_application 2023-05-30 21:31:37.828063 coordination.COORDINATOR.start() 2023-05-30 21:31:37.828068 File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/coordination.py", line 86, in start 2023-05-30 21:31:37.828071 self.coordinator.start(start_heart=True) 2023-05-30 21:31:37.828075 File "/var/lib/kolla/venv/lib/python3.6/site-packages/tooz/coordination.py", line 689, in start 2023-05-30 21:31:37.828078 super(CoordinationDriverWithExecutor, self).start(start_heart) 2023-05-30 21:31:37.828083 File "/var/lib/kolla/venv/lib/python3.6/site-packages/tooz/coordination.py", line 426, in start 2023-05-30 21:31:37.828086 self._start() 2023-05-30 21:31:37.828090 File "/var/lib/kolla/venv/lib/python3.6/site-packages/tooz/drivers/etcd3gw.py", line 224, in _start 2023-05-30 21:31:37.828093 self._membership_lease = self.client.lease(self.membership_timeout) 2023-05-30 21:31:37.828098 File "/var/lib/kolla/venv/lib/python3.6/site-packages/etcd3gw/client.py", line 122, in lease 2023-05-30 21:31:37.828111 json={"TTL": ttl, "ID": 0}) 2023-05-30 21:31:37.828116 File "/var/lib/kolla/venv/lib/python3.6/site-packages/etcd3gw/client.py", line 88, in post 2023-05-30 21:31:37.828123 resp.reason 2023-05-30 21:31:37.828154 etcd3gw.exceptions.ConnectionTimeoutError: Gateway Time-out
All other containers are working just fine. Even the cinder_scheduler container works fine.
So far I have tried the following:
remove the cinder containers including its volume from one control node
mariadb_recovery
Reboot all control nodes.
kolla-ansible reconfigure --tags cinder,nova
Any help is highly appreciated.
Cheers,
Oliver