Hi Team ,

We have an 'Openstack on Ansible' Private cloud environment with 3 controllers & 35 Compute host nodes. Age of this environment is around 4 yrs.

DISTRIB_RELEASE: 21.2.1
DISTRIB_CODENAME: Ussuri
DISTRIB_DESCRIPTION: Openstack-Ansible

Top of this Linux containers are there where the services are running inside such as neutron, galera, horizon, keystone, etcd, nova-api, utility etc for all 3 controllers.

Current Set Up:
We find in haproxy config file only controller2 & controller3 are configured.
So the galera container along with Mariadb Services are running fine within controller2(Primary) & controller3.
But Galera container in controller1 is stopped & mariadb service not running here.

Issue Started with:
All of suddent we faced, user not able to log in dashboard 5 days back. The dashboard page is not getting loaded.


Action Taken:
We find Maria-DB in controller2 is non-primary & DB on controller-3 is down.
Made Maria-DB in controller2 primary & DB on controller-3 up which resulted to load the page & log in.

Current Issue:
1) But the Dashboard page is too slow to work. Sometime throwing 504-Timed out error
2) VM Console(VNC) not working for any VM
3) NOt able to create any VM(Showing 'Scheduling' continuously)
4) 25/35 Hypervisons are showing down in Dashboard within 'Hypervisor list' tab but those are up as per CLI
5) Not able to create any volume


Request you to go through the Problem description & Can anybody help me out providing the solution for the same?

Pl do intimate me if any more info is required.


Regards,
Sudeb Ghosh
sudeb_ece@yahoo.co.in















BR//
Sudeb Ghosh
7044064878
9332034788