[kolla] [mariadb] Backups failing if they take over an hour
Were using the mariadb_backup command of kolla-ansible to backup the database. We're doing daily, full backups. It's been working fine until recently. Now we are getting this error: 2024-04-03 21:16:33 0 [Note] InnoDB: Read redo log up to LSN=317647271424 2024-04-03 21:16:48 0 [Note] InnoDB: Read redo log up to LSN=317647424512 2024-04-03 21:17:03 0 [Note] InnoDB: Read redo log up to LSN=317647586304 2024-04-03 21:17:18 0 [Note] InnoDB: Read redo log up to LSN=317647733248 [01] 2024-04-03 21:17:23 ...done [00] FATAL ERROR: 2024-04-03 21:17:23 failed to execute query SET SESSION lock_wait_timeout=31536000: Server has gone away It started as soon as the DB backup took more than 1 hour. In our other environments where the backup is < 1hr, it works fine. There are no errors in the logs when this happens, other then the above output from mariabackup Any suggestions? Thanks, Chris
If you have any 60-minute timeout values configured for mariadb, that would be a likely culprit. Also ISTR that haproxy timeout values can be a factor. On Thursday, April 4, 2024 at 10:24:12 AM EDT, Chris DiLorenzo <cdilorenzo@gmail.com> wrote: Were using the mariadb_backup command of kolla-ansible to backup the database. We're doing daily, full backups. It's been working fine until recently. Now we are getting this error: 2024-04-03 21:16:33 0 [Note] InnoDB: Read redo log up to LSN=3176472714242024-04-03 21:16:48 0 [Note] InnoDB: Read redo log up to LSN=3176474245122024-04-03 21:17:03 0 [Note] InnoDB: Read redo log up to LSN=3176475863042024-04-03 21:17:18 0 [Note] InnoDB: Read redo log up to LSN=317647733248[01] 2024-04-03 21:17:23 ...done [00] FATAL ERROR: 2024-04-03 21:17:23 failed to execute query SET SESSION lock_wait_timeout=31536000: Server has gone away It started as soon as the DB backup took more than 1 hour. In our other environments where the backup is < 1hr, it works fine. There are no errors in the logs when this happens, other then the above output from mariabackup Any suggestions? Thanks,Chris
participants (2)
-
Albert Braden
-
Chris DiLorenzo