[dev][swift][account-reaper] Account with sharded bucket
Hi, I'm using swift-account-reaper for accounts with buckets in an unsharded state, and it looks fine, wiping the account. But when I try to wipe an account with sharded buckets, I encounter some problematic scenarios. First, when deleting the account, account-reaper lists objects from the sharded bucket, an empty list is returned from *direct_get_container* function, so the reaper interprets it as an empty bucket, and the delete fails with a 409 Conflict because objects exist inside the bucket. My second thought was to delete the account that holds shard buckets (i.e. .shards_AUTH_<project_id>) so the reaper will wipe this account. Then, with all buckets empty, they'll be deleted from the principal account (AUTH_<project_id>). But in this case, when the reaper tries to delete objects inside this account, all objects are not found because the object path used is "/.shards_AUTH_<project_id>/<bucket>-<hash>- <timestamp>-<shard_index>/<key>" instead of "/AUTH_<project_id>/ <bucket>/<key>". Am I doing something wrong in trying to delete this account? Or does account-reaper, in fact, not support sharded buckets? I'm exploring some patches to make it work, and I can volunteer to work on this fix if necessary. Thanks! -- _‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’._ * **‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
On Tue, 4 Jun 2024 14:29:23 -0300 Thiago De Moraes Teixeira <teixeira.thiago@luizalabs.com> wrote:
First, when deleting the account, account-reaper lists objects from the sharded bucket, an empty list is returned from *direct_get_container* function, so the reaper interprets it as an empty bucket, and the delete fails with a 409 Conflict because objects exist inside the bucket.
That looks like a bug. I'm astonished that nobody has encountered it yet, but users of direct_get_container must check for retuned shards. There's some sample code in the object watcher, https://opendev.org/openstack/swift/src/branch/master/swift/obj/watchers/dar...
... But in this case, when the reaper tries to delete objects inside this account, all objects are not found because the object path used is "/.shards_AUTH_<project_id>/<bucket>-<hash>- <timestamp>-<shard_index>/<key>" instead of "/AUTH_<project_id>/ <bucket>/<key>".
Not sure what to say about that. Either way, I think the best course of action is to file a bug in Launchpad. This may need some not entirely trivial work. -- Pete
Sorry for the delayed response; for some reason, this email ended up in my spam folder. I have opened a bug report on Launchpad ( https://bugs.launchpad.net/swift/+bug/2070397). Regarding the handling that the dark_data watcher performs for shards in direct_get_container, this approach is almost identical to the workaround I implemented to ensure the account-reaper functions as expected. I have attached the patch file to the bug report. Some variable names might be confusing at this point, but they can be adjusted as needed.
This may require some non-trivial work.
Do you have any concerns about this approach? I'm trying to think about potential scenarios. Em qui., 20 de jun. de 2024 às 01:48, Pete Zaitcev <zaitcev@redhat.com> escreveu:
On Tue, 4 Jun 2024 14:29:23 -0300 Thiago De Moraes Teixeira <teixeira.thiago@luizalabs.com> wrote:
First, when deleting the account, account-reaper lists objects from the sharded bucket, an empty list is returned from *direct_get_container* function, so the reaper interprets it as an empty bucket, and the delete fails with a 409 Conflict because objects exist inside the bucket.
That looks like a bug. I'm astonished that nobody has encountered it yet, but users of direct_get_container must check for retuned shards. There's some sample code in the object watcher,
https://opendev.org/openstack/swift/src/branch/master/swift/obj/watchers/dar...
... But in this case, when the reaper tries to delete objects inside this account, all objects are not found because the object path used is "/.shards_AUTH_<project_id>/<bucket>-<hash>- <timestamp>-<shard_index>/<key>" instead of "/AUTH_<project_id>/ <bucket>/<key>".
Not sure what to say about that.
Either way, I think the best course of action is to file a bug in Launchpad. This may need some not entirely trivial work.
-- Pete
-- _‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’._ * **‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
On Tue, 2024-06-25 at 14:41 -0300, Thiago De Moraes Teixeira wrote:
Sorry for the delayed response; for some reason, this email ended up in my spam folder.
I have opened a bug report on Launchpad (https://bugs.launchpad.net/swift/+bug/2070397).
Looks good.
Do you have any concerns about this approach? I'm trying to think about potential scenarios.
I think it's correct, but it took me a long time to push the fix through for the watcher. Let's see to it that we're aren't doing the same in the reaper. -- P
I can improve the workaround code and submit it to OpenDev for review if this would be helpful. I appreciate your feedback. Em ter., 25 de jun. de 2024 às 15:27, Pete Zaitcev <zaitcev@redhat.com> escreveu:
On Tue, 2024-06-25 at 14:41 -0300, Thiago De Moraes Teixeira wrote:
Sorry for the delayed response; for some reason, this email ended up in my spam folder.
I have opened a bug report on Launchpad (https://bugs.launchpad.net/swift/+bug/2070397).
Looks good.
Do you have any concerns about this approach? I'm trying to think about potential scenarios.
I think it's correct, but it took me a long time to push the fix through for the watcher. Let's see to it that we're aren't doing the same in the reaper.
-- P
-- _‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’._ * **‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
Tim (our PTL) saw the bug that you filed and we discussed it at a meeting today. Take a look, it's at the beginning of the log: https://meetings.opendev.org/meetings/swift/2024/swift.2024-06-26-21.00.log.... If you could make up the probe test, or file a review using the snippet that you attached to the bug, any of it would help. -- Pete On Tue, 2024-06-25 at 15:50 -0300, Thiago De Moraes Teixeira wrote:
I can improve the workaround code and submit it to OpenDev for review if this would be helpful.
I appreciate your feedback.
Em ter., 25 de jun. de 2024 às 15:27, Pete Zaitcev <zaitcev@redhat.com> escreveu:
On Tue, 2024-06-25 at 14:41 -0300, Thiago De Moraes Teixeira wrote:
Sorry for the delayed response; for some reason, this email ended up in my spam folder.
I have opened a bug report on Launchpad (https://bugs.launchpad.net/swift/+bug/2070397).
Looks good.
Do you have any concerns about this approach? I'm trying to think about potential scenarios.
I think it's correct, but it took me a long time to push the fix through for the watcher. Let's see to it that we're aren't doing the same in the reaper.
-- P
‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’. ‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.
participants (2)
-
Pete Zaitcev
-
Thiago De Moraes Teixeira