Hi Philipp, This sounds to me like an issue with routing, when you're trying to reach internal network through the default gateway (or your external_vip is in the same subnet as the internal_vip?). In case your internal and external VIPs are in the same network, you can just override variable `galera_monitoring_allowed_source` [1] and define it to the valid list of IPs that you expect to check for cluster status [1] https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/... 12.02.2021, 14:37, "Philipp Wörner" <philipp.woerner@dhbw-mannheim.de>:

Dear all,

unfortunately we are facing an problem while setting up Openstack (described at the end of the mail).

Last year we had no issue with our configuration.

I found the problem why the playbook stops and a temporary workaround.

But I don’t know the root cause, maybe you can help me with some advice.

Thank you in advance!

Have a sunny weekend and best regards,

Philipp

Where setup-openstack.yml stops:

TASK [os_keystone : Check current state of Keystone DB] **************************************************************************************************************************************************************************************

fatal: [infra1_keystone_container-01c233df]: FAILED! => {"changed": true, "cmd": ["/openstack/venvs/keystone-21.2.3.dev4/bin/keystone-manage", "db_sync", "--check"], "delta": "0:01:42.166790", "end": "2021-02-11 13:01:06.282388", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2021-02-11 12:59:24.115598", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

Reason:

This is caused because HAProxy thinks the service is not available… caused by the health check against port 9200.

If I test the cluster-check-service manually, I see the external_vip isn’t allowed.

root@bc1bl11:/home/ubuntu# telnet -b <internal_vip> 192.168.110.235 9200

Trying 192.168.110.235...

Connected to 192.168.110.235.

Escape character is '^]'.

HTTP/1.1 200 OK

Content-Type: text/plain

Connection: close

Content-Length: 40

Percona XtraDB Cluster Node is synced.

Connection closed by foreign host.

root@bc1bl11:/home/ubuntu# telnet -b <external_vip> 192.168.110.235 9200

Trying 192.168.110.235...

Connected to 192.168.110.235.

Escape character is '^]'.

Connection closed by foreign host.

Workaround:

After manually modifying the service-configuration and adding the external_vip to the whitelist everything works and the OSA-playbook succeeds as well:

root@infra1-galera-container-492e1206:/# cat /etc/xinetd.d/mysqlchk

# default: on

# description: mysqlchk

# Ansible managed

service mysqlchk

{

        disable = no

        flags = REUSE

        socket_type = stream

        port = 9200

        wait = no

        user = nobody

        server = /usr/local/bin/clustercheck

        log_on_failure += USERID

                only_from = 192.168.110.200 192.168.110.235 192.168.110.211 127.0.0.1

                per_source = UNLIMITED

}

Question:

I am wondering now why haproxy uses the external_vip to check the mysql-service and why I am facing this problem now… because last year everything was fine with our configuration.

We just moved the external_vip from the NIC to the bridge in the netplan-config and the external_vip is now in the same network as the internal_vip.

Here is also a snipped of our haproxy config:

root@bc1bl11:/home/ubuntu# cat /etc/haproxy/haproxy.cfg

# Ansible managed

global

        log /dev/log local0

        chroot /var/lib/haproxy

        user haproxy

        group haproxy

        daemon

        maxconn 4096

        stats socket /var/run/haproxy.stat level admin mode 600

                ssl-default-bind-options force-tlsv12

        tune.ssl.default-dh-param 2048

defaults

        log global

        option dontlognull

        option redispatch

        option forceclose

        retries 3

        timeout client 50s

        timeout connect 10s

        timeout http-request 5s

        timeout server 50s

        maxconn 4096

…

frontend galera-front-1

    bind 192.168.110.211:3306

    option tcplog

    timeout client 5000s

    acl white_list src 127.0.0.1/8 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8

    tcp-request content accept if white_list

    tcp-request content reject

    mode tcp

    default_backend galera-back

backend galera-back

    mode tcp

    balance leastconn

    timeout server 5000s

    stick store-request src

    stick-table type ip size 256k expire 30m

    option tcplog

    option httpchk HEAD / HTTP/1.0\r\nUser-agent:\ osa-haproxy-healthcheck

   # server infra1_galera_container-492e1206 192.168.110.235:3306

   server infra1_galera_container-492e1206 192.168.110.235:3306 check port 9200 inter 12000 rise 1 fall 1

--  Kind Regards, Dmitriy Rabotyagov