[kolla] [cinder-backup] max operations and workers problems
good morning, using all-in-one ubuntt 22 lts kolla openstack 2024.1 version. cinder.conf " [DEFAULT] ... backup_max_operations = 1 backup_workers = 1 ... " despite this, the cinder-backup container is doing more backups of more istance volumes...why? thank you.
Hi Garcetto, Can you provide the details of how you determined that cinder-backup is doing more than 1 operation? For example, if you launch 10 backup operations, it will show 10 backups being created in ``cinder backup-list`` but it does not mean they are being executed at the same time. There is a locking mechanism that holds the other requests until one request completes. So in terms of memory consumption, there will always be 1 backup operation consuming the memory (which is the purpose of this config option i.e. to limit memory consumption). But that's just an assumption from my side, if there is a real issue somewhere, it would be good to report and we can work on fixing it. Thanks Rajat Dhasmana On Wed, Oct 2, 2024 at 3:36 PM garcetto <garcetto@gmail.com> wrote:
good morning, using all-in-one ubuntt 22 lts kolla openstack 2024.1 version.
cinder.conf " [DEFAULT] ... backup_max_operations = 1 backup_workers = 1 ... "
despite this, the cinder-backup container is doing more backups of more istance volumes...why?
thank you.
because i am monitoring the backup repo (in this case s3 bucket) and i see more backup files written in parallel in different subdirs (so different volumes), so different backups are running. and also monitoring mem usage on cinder-backup shows that more files are backed up [image: image.png] On Thu, Oct 3, 2024 at 9:59 AM Rajat Dhasmana <rdhasman@redhat.com> wrote:
Hi Garcetto,
Can you provide the details of how you determined that cinder-backup is doing more than 1 operation? For example, if you launch 10 backup operations, it will show 10 backups being created in ``cinder backup-list`` but it does not mean they are being executed at the same time. There is a locking mechanism that holds the other requests until one request completes. So in terms of memory consumption, there will always be 1 backup operation consuming the memory (which is the purpose of this config option i.e. to limit memory consumption).
But that's just an assumption from my side, if there is a real issue somewhere, it would be good to report and we can work on fixing it.
Thanks Rajat Dhasmana
On Wed, Oct 2, 2024 at 3:36 PM garcetto <garcetto@gmail.com> wrote:
good morning, using all-in-one ubuntt 22 lts kolla openstack 2024.1 version.
cinder.conf " [DEFAULT] ... backup_max_operations = 1 backup_workers = 1 ... "
despite this, the cinder-backup container is doing more backups of more istance volumes...why?
thank you.
Hi Garcetto, Can you provide the following details about your environment: 1. deployment tool - I'm assuming it's kolla ansible 2. cinder.conf file (the DEFAULT section should suffice to extract backup related configuration) 3. cinder-backup DEBUG logs of concurrent backup create operations If we set, backup_max_operations = 1 backup_workers = 1 then we use an eventlet semaphore to lock the backup creation (critical section) in the manager that should not cause an issue if you are using a single backup service in your environment. I'm assuming it could be something in the way things are deployed that are causing multiple backup requests to not honour the semaphore i.e. multiple cinder-backup services running in your environment or something related. Thanks Rajat Dhasmana On Thu, Oct 3, 2024 at 5:01 PM garcetto <garcetto@gmail.com> wrote:
because i am monitoring the backup repo (in this case s3 bucket) and i see more backup files written in parallel in different subdirs (so different volumes), so different backups are running. and also monitoring mem usage on cinder-backup shows that more files are backed up
[image: image.png]
On Thu, Oct 3, 2024 at 9:59 AM Rajat Dhasmana <rdhasman@redhat.com> wrote:
Hi Garcetto,
Can you provide the details of how you determined that cinder-backup is doing more than 1 operation? For example, if you launch 10 backup operations, it will show 10 backups being created in ``cinder backup-list`` but it does not mean they are being executed at the same time. There is a locking mechanism that holds the other requests until one request completes. So in terms of memory consumption, there will always be 1 backup operation consuming the memory (which is the purpose of this config option i.e. to limit memory consumption).
But that's just an assumption from my side, if there is a real issue somewhere, it would be good to report and we can work on fixing it.
Thanks Rajat Dhasmana
On Wed, Oct 2, 2024 at 3:36 PM garcetto <garcetto@gmail.com> wrote:
good morning, using all-in-one ubuntt 22 lts kolla openstack 2024.1 version.
cinder.conf " [DEFAULT] ... backup_max_operations = 1 backup_workers = 1 ... "
despite this, the cinder-backup container is doing more backups of more istance volumes...why?
thank you.
participants (2)
-
garcetto
-
Rajat Dhasmana