I’m experimenting with Galera in my Rocky openstack-ansible dev cluster, and I’m finding that the default haproxy config values don’t seem to work. Finding the correct values is a lot of work. For example, I spent this morning experimenting with different values for “timeout  client” in /etc/haproxy/haproxy.cfg. The default is 1m, and with the default set I see this error in /var/log/nova/nova-scheduler.log on the controllers:

2020-01-17 13:54:26.059 443358 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)

There are several timeout values in /etc/haproxy/haproxy.cfg. These are the values we started with:

stats timeout 30s

timeout  http-request 10s

timeout  queue 1m

timeout  connect 10s

timeout  client 1m

timeout  server 1m

timeout  check 10s

At first I changed them all to 30m. This stopped the “Lost connection” error in nova-scheduler.log. Then, one at a time, I changed them back to the default. When I got to “timeout  client” I found that setting it back to 1m caused the errors to start again. I changed it back and forth and found that 4 minutes causes errors, and 6m stops them, so I left it at 6m.

These are my active variables:

root@us01odc-dev2-ctrl1:/etc/mysql# mysql -e 'show variables;'|grep timeout

connect_timeout 20

deadlock_timeout_long   50000000

deadlock_timeout_short  10000

delayed_insert_timeout  300

idle_readonly_transaction_timeout       0

idle_transaction_timeout        0

idle_write_transaction_timeout  0

innodb_flush_log_at_timeout     1

innodb_lock_wait_timeout        50

innodb_rollback_on_timeout      OFF

interactive_timeout     28800

lock_wait_timeout       86400

net_read_timeout        30

net_write_timeout       60

rpl_semi_sync_master_timeout    10000

rpl_semi_sync_slave_kill_conn_timeout   5

slave_net_timeout       60

thread_pool_idle_timeout        60

wait_timeout    3600

So it looks like the value of “timeout  client” in haproxy.cfg needs to match or exceed the value of “wait_timeout” in mysql. Also in nova.conf I see “#connection_recycle_time = 3600” – I need to experiment to see how that value interacts with the timeouts in the other config files.

Is this the best way to find the correct config values? It seems like there should be a document that talks about these timeouts and how to set them (or maybe more generally how the different timeout settings in the various config files interact). Does that document exist? If not, maybe I could write one, since I have to figure out the correct values anyway.