Open Stack

I'm experimenting with Galera in my Rocky openstack-ansible dev cluster, and I'm finding that the default haproxy config values don't seem to work. Finding the correct values is a lot of work. For example, I spent this morning experimenting with different values for "timeout  client" in /etc/haproxy/haproxy.cfg. The default is 1m, and with the default set I see this error in /var/log/nova/nova-scheduler.log on the controllers:

2020-01-17 13:54:26.059 443358 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)

There are several timeout values in /etc/haproxy/haproxy.cfg. These are the values we started with:

stats timeout 30s
timeout  http-request 10s
timeout  queue 1m
timeout  connect 10s
timeout  client 1m
timeout  server 1m
timeout  check 10s

At first I changed them all to 30m. This stopped the "Lost connection" error in nova-scheduler.log. Then, one at a time, I changed them back to the default. When I got to "timeout  client" I found that setting it back to 1m caused the errors to start again. I changed it back and forth and found that 4 minutes causes errors, and 6m stops them, so I left it at 6m.

These are my active variables:

root at us01odc-dev2-ctrl1:/etc/mysql# mysql -e 'show variables;'|grep timeout
connect_timeout 20
deadlock_timeout_long   50000000
deadlock_timeout_short  10000
delayed_insert_timeout  300
idle_readonly_transaction_timeout       0
idle_transaction_timeout        0
idle_write_transaction_timeout  0
innodb_flush_log_at_timeout     1
innodb_lock_wait_timeout        50
innodb_rollback_on_timeout      OFF
interactive_timeout     28800
lock_wait_timeout       86400
net_read_timeout        30
net_write_timeout       60
rpl_semi_sync_master_timeout    10000
rpl_semi_sync_slave_kill_conn_timeout   5
slave_net_timeout       60
thread_pool_idle_timeout        60
wait_timeout    3600

So it looks like the value of "timeout  client" in haproxy.cfg needs to match or exceed the value of "wait_timeout" in mysql. Also in nova.conf I see "#connection_recycle_time = 3600" - I need to experiment to see how that value interacts with the timeouts in the other config files.

Is this the best way to find the correct config values? It seems like there should be a document that talks about these timeouts and how to set them (or maybe more generally how the different timeout settings in the various config files interact). Does that document exist? If not, maybe I could write one, since I have to figure out the correct values anyway.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200117/f65352d4/attachment.html>