This is the config in the haproxy:
backend rabbitmq_mgmt-back
mode http
balance leastconn
stick-table type ipv6 size 256k expire 10s store http_err_rate(10s)
http-request track-sc0 src
http-request deny deny_status 429 if { sc_http_err_rate(0) gt 20 } !{ src
192.168.0.0/16 } !{ src
172.16.0.0/12 } !{ src
10.0.0.0/8 }
option forwardfor
option httpchk HEAD / HTTP/1.0\r\nUser-agent:\ osa-haproxy-healthcheck
server rabbit_mq_container-962bf7ee
10.91.31.227:15672 check port 15672 inter 12000 rise 3 fall 3 ssl check-ssl verify none
server rabbit_mq_container-d800c10f
10.91.32.111:15672 check port 15672 inter 12000 rise 3 fall 3 ssl check-ssl verify none
server rabbit_mq_container-dc9fb45e
10.91.32.239:15672 check port 15672 inter 12000 rise 3 fall 3 ssl check-ssl verify none
I dont see any TLS connection here as well:
----
# Management plugin configuration
management.ssl.ip = 10.90.31.206
management.ssl.port = 15672
management.ssl.certfile = /etc/rabbitmq/rabbitmq.pem
management.ssl.keyfile = /etc/rabbitmq/rabbitmq.key
management.ssl.honor_cipher_order = true
management.ssl.honor_ecc_order = true
management.ssl.client_renegotiation = false
management.ssl.secure_renegotiate = true
management.ssl.versions.1 = tlsv1.2
management.ssl.ciphers.1 = ECDHE-ECDSA-AES256-GCM-SHA384
management.ssl.ciphers.2 = ECDHE-RSA-AES256-GCM-SHA384
management.ssl.ciphers.3 = ECDH-ECDSA-AES256-GCM-SHA384
management.ssl.ciphers.4 = ECDH-RSA-AES256-GCM-SHA384
management.ssl.ciphers.5 = DHE-RSA-AES256-GCM-SHA384
management.ssl.ciphers.6 = DHE-DSS-AES256-GCM-SHA384
management.ssl.ciphers.7 = ECDHE-ECDSA-AES128-GCM-SHA256
management.ssl.ciphers.8 = ECDHE-RSA-AES128-GCM-SHA256
management.ssl.ciphers.9 = ECDH-ECDSA-AES128-GCM-SHA256
management.ssl.ciphers.10 = ECDH-RSA-AES128-GCM-SHA256
management.ssl.ciphers.11 = DHE-RSA-AES128-GCM-SHA256
management.ssl.ciphers.12 = DHE-DSS-AES128-GCM-SHA256
management.ssl.verify = verify_none
management.ssl.fail_if_no_peer_cert = false
So how is the health check succeeding ?
So, does this port 15671, is it not the same as TLS ? If so, why are we using it as backend for health check ? It is being generated by openstack-ansible automatically..
Actually what I've spotted is that you're trying to do CURL through basic TCP, while HAProxy does check-ssl, which means there's likely a TLS connection with the backend. And also 15671 port is SSL port for RabbitMQ management.
But I wouldn't expect "Connection refused" though - more an empty reply or some protocol mismatch...
Also can you kindly clarify - 172.21.152.187 is IP/port of the backend or haproxy internal VIP? I assume the backend? What do you receive when you try to curl that haproxy frontend then?
Hi there,
From the configuration snippet, I see you have "inter 12000" (12 seconds between health checks), and "fall 3" configured. This means that HAProxy will not consider the backend member failed until after roughly 36 seconds. Because of that "fall 3" (which is also the default) configuration, HAProxy will consider the member "up", even though it may not be, during this 36 second window.
Maybe this is the behavior you are observing?
Michael
Hi,
We have the latest version of openstack setup. The rabbitmq backend health check is getting successful even though the curl to its ssl url is failing as shown below:
----
backend rabbitmq_mgmt-back
mode http
balance leastconn
stick-table type ipv6 size 256k expire 10s store http_err_rate(10s)
http-request track-sc0 src
http-request deny deny_status 429 if { sc_http_err_rate(0) gt 20 } !{ src
192.168.0.0/16 } !{ src
172.16.0.0/12 } !{ src
10.0.0.0/8 }
option forwardfor
option httpchk HEAD / HTTP/1.0\r\nUser-agent:\ osa-haproxy-healthcheck
server rabbit_mq_container-527affef <ip>:15671 check port 15671 inter 12000 rise 3 fall 3 ssl check-ssl verify none
server rabbit_mq_container-68b39094 <IP>:15671 check port 15671 inter 12000 rise 3 fall 3 ssl check-ssl verify none
server rabbit_mq_container-83661288 <IP>:15671 check port 15671 inter 12000 rise 3 fall 3 ssl check-ssl verify none
------
But the ssl curl to its ip and port 15671 is failing:
| curl -vvv 172.21.152.187:15671
* Trying 172.21.152.187:15671...
* TCP_NODELAY set
* connect to 172.21.152.187 port 15671 failed: Connection refused
* Failed to connect to 172.21.152.187 port 15671: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 172.21.152.187 port 15671: Connection refused
|
How then, the haproxy is still able to detect it as green and status ok in the haproxy stats page ?