If you have any 60-minute timeout values configured for mariadb, that would be a likely culprit. Also ISTR that haproxy timeout values can be a factor.
On Thursday, April 4, 2024 at 10:24:12 AM EDT, Chris DiLorenzo <cdilorenzo@gmail.com> wrote:


Were using the mariadb_backup command of kolla-ansible to backup the database.  We're doing daily, full backups.  It's been working fine until recently.  Now we are getting this error:

2024-04-03 21:16:33 0 [Note] InnoDB: Read redo log up to LSN=317647271424
2024-04-03 21:16:48 0 [Note] InnoDB: Read redo log up to LSN=317647424512
2024-04-03 21:17:03 0 [Note] InnoDB: Read redo log up to LSN=317647586304
2024-04-03 21:17:18 0 [Note] InnoDB: Read redo log up to LSN=317647733248
[01] 2024-04-03 21:17:23 ...done [00] FATAL ERROR: 2024-04-03 21:17:23 failed to execute query SET SESSION lock_wait_timeout=31536000: Server has gone away

It started as soon as the DB backup took more than 1 hour.  In our other environments where the backup is < 1hr, it works fine.  There are no errors in the logs when this happens, other then the above output from mariabackup

Any suggestions?

Thanks,
Chris