[nova][cinder][glance][manila][masakari][tacker][oslo] Configurable soft-delete
👋 I'm planning on bringing this up in the nova rooms at the PTG in a few weeks, but I'm also raising it here since this potentially affects other service projects and I can't attend all of those room :) Many projects use the concept of "soft delete" in their database models. A soft deletable model typically has two additional columns, 'deleted' and 'deleted_at'. When deleting such a model, instead of actually deleting the database row (i.e. 'DELETE FROM table WHERE condition'), we set 'deleted' to 'True' and populate the 'deleted_at' column. This is helpful for auditing purposes (e.g. you can inspect all resources ever created, even after they've been "deleted") but bad for database performance (your tables can grow without bound). To work around the performance issues, most projects implement some kind of archive or purge command that will allow operators to periodically clean up these deleted resources. However, at least in nova, we've long since come to the conclusion that soft deleting isn't as useful as initially suspected and the need to run these commands is additional work for no benefit. We've moved toward not using it for all new models. With this said, it's going to be difficult to get away from soft-delete quickly. Not only are there database migrations involved, but operators will need to rework their tooling to adapt to a new, no-soft-delete world. As such, I'd like to propose a half-way measure of making soft-delete configurable. To do this, I'd like to add a new flag in oslo.db, '[database] enable_soft_delete'. When set to 'False' anyone using the 'SoftDeleteMixin' from oslo.db would see these models hard deleted rather than soft deleted when calling 'soft_delete'. This would avoid the need for operators to run the various project-specific purge tooling. The RFC patch for this is available for review [1]. I can also do this on a project-specific basis and have proposed a similar patch for nova [2], however, doing it in oslo.db means every project that uses 'SoftDeleteMixin' in their models will get this for free. Projects that don't (glance, cinder) can switch to using this mixin and also get it for free. As noted above, I intend to discuss this in the nova room at the PTG, but I'd be interested in people's thoughts ahead of time. Do you think this is a good idea? Should we proceed with it? Perhaps there are there better ways to do this? Let me know! Cheers, Stephen [1] https://review.opendev.org/c/openstack/oslo.db/+/860407 [2] https://review.opendev.org/c/openstack/nova/+/860401
Not having soft delete in the database is really quite bad for operators and it's not about tooling, but it's about audit purposes. If take nova as example, this also means that once server is deleted, event log will be also wiped with no way to see who and when has performed delete action. And that is really used feature, as we got requests like "why my VM has disappeared" at very least one a week. For other services having deleted_at at least points to the datetime where to search in the logs. At the same time I don't see any issue in having soft delete. It's just a matter of one systemd-timer, and too concerned about performance can set it to 1 day, thus almost no impact on db performance. So from operator perspective I can say this is very valuable feature and I personally do struggle regularly with neutron services where it's absent. And I would hate this to disappear at all, as it would be really a nightmare. ср, 5 окт. 2022 г., 14:48 Stephen Finucane <stephenfin@redhat.com>:
👋
I'm planning on bringing this up in the nova rooms at the PTG in a few weeks, but I'm also raising it here since this potentially affects other service projects and I can't attend all of those room :)
Many projects use the concept of "soft delete" in their database models. A soft deletable model typically has two additional columns, 'deleted' and 'deleted_at'. When deleting such a model, instead of actually deleting the database row (i.e. 'DELETE FROM table WHERE condition'), we set 'deleted' to 'True' and populate the 'deleted_at' column. This is helpful for auditing purposes (e.g. you can inspect all resources ever created, even after they've been "deleted") but bad for database performance (your tables can grow without bound). To work around the performance issues, most projects implement some kind of archive or purge command that will allow operators to periodically clean up these deleted resources. However, at least in nova, we've long since come to the conclusion that soft deleting isn't as useful as initially suspected and the need to run these commands is additional work for no benefit. We've moved toward not using it for all new models.
With this said, it's going to be difficult to get away from soft-delete quickly. Not only are there database migrations involved, but operators will need to rework their tooling to adapt to a new, no-soft-delete world. As such, I'd like to propose a half-way measure of making soft-delete configurable. To do this, I'd like to add a new flag in oslo.db, '[database] enable_soft_delete'. When set to 'False' anyone using the 'SoftDeleteMixin' from oslo.db would see these models hard deleted rather than soft deleted when calling 'soft_delete'. This would avoid the need for operators to run the various project-specific purge tooling. The RFC patch for this is available for review [1]. I can also do this on a project-specific basis and have proposed a similar patch for nova [2], however, doing it in oslo.db means every project that uses 'SoftDeleteMixin' in their models will get this for free. Projects that don't (glance, cinder) can switch to using this mixin and also get it for free.
As noted above, I intend to discuss this in the nova room at the PTG, but I'd be interested in people's thoughts ahead of time. Do you think this is a good idea? Should we proceed with it? Perhaps there are there better ways to do this? Let me know!
Cheers, Stephen
[1] https://review.opendev.org/c/openstack/oslo.db/+/860407 [2] https://review.opendev.org/c/openstack/nova/+/860401
Hey, If this is there mostly for audit purposes then I guess a more efficient solution is to introduce an audit table which will have no performance impact on the "current state" of service. Audit records are also never updated means performance here is relatively straight forward. Audit may become a "feature" with enable switch. This should be a better solution rather then letting db permanently grow and forcing admins to constantly "fight" against it. This also gives a much cleaner audit experience. Actually this is also not a new approach and is being followed in many places (i.e. auditd) Regards, Artem ---- typed from mobile, auto-correct typos assumed ---- On Thu, Oct 6, 2022, 07:27 Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Not having soft delete in the database is really quite bad for operators and it's not about tooling, but it's about audit purposes.
If take nova as example, this also means that once server is deleted, event log will be also wiped with no way to see who and when has performed delete action. And that is really used feature, as we got requests like "why my VM has disappeared" at very least one a week.
For other services having deleted_at at least points to the datetime where to search in the logs.
At the same time I don't see any issue in having soft delete. It's just a matter of one systemd-timer, and too concerned about performance can set it to 1 day, thus almost no impact on db performance.
So from operator perspective I can say this is very valuable feature and I personally do struggle regularly with neutron services where it's absent. And I would hate this to disappear at all, as it would be really a nightmare.
ср, 5 окт. 2022 г., 14:48 Stephen Finucane <stephenfin@redhat.com>:
👋
I'm planning on bringing this up in the nova rooms at the PTG in a few weeks, but I'm also raising it here since this potentially affects other service projects and I can't attend all of those room :)
Many projects use the concept of "soft delete" in their database models. A soft deletable model typically has two additional columns, 'deleted' and 'deleted_at'. When deleting such a model, instead of actually deleting the database row (i.e. 'DELETE FROM table WHERE condition'), we set 'deleted' to 'True' and populate the 'deleted_at' column. This is helpful for auditing purposes (e.g. you can inspect all resources ever created, even after they've been "deleted") but bad for database performance (your tables can grow without bound). To work around the performance issues, most projects implement some kind of archive or purge command that will allow operators to periodically clean up these deleted resources. However, at least in nova, we've long since come to the conclusion that soft deleting isn't as useful as initially suspected and the need to run these commands is additional work for no benefit. We've moved toward not using it for all new models.
With this said, it's going to be difficult to get away from soft-delete quickly. Not only are there database migrations involved, but operators will need to rework their tooling to adapt to a new, no-soft-delete world. As such, I'd like to propose a half-way measure of making soft-delete configurable. To do this, I'd like to add a new flag in oslo.db, '[database] enable_soft_delete'. When set to 'False' anyone using the 'SoftDeleteMixin' from oslo.db would see these models hard deleted rather than soft deleted when calling 'soft_delete'. This would avoid the need for operators to run the various project-specific purge tooling. The RFC patch for this is available for review [1]. I can also do this on a project-specific basis and have proposed a similar patch for nova [2], however, doing it in oslo.db means every project that uses 'SoftDeleteMixin' in their models will get this for free. Projects that don't (glance, cinder) can switch to using this mixin and also get it for free.
As noted above, I intend to discuss this in the nova room at the PTG, but I'd be interested in people's thoughts ahead of time. Do you think this is a good idea? Should we proceed with it? Perhaps there are there better ways to do this? Let me know!
Cheers, Stephen
[1] https://review.opendev.org/c/openstack/oslo.db/+/860407 [2] https://review.opendev.org/c/openstack/nova/+/860401
Oh, yes, this is a good alternative. Actually I was thinking about smth like "openstack <resource> event list" (as only nova does have that) for quite a while, but without having the resources to lead and help out in implementation across projects I didn't dare to raise this topic. But it's probably high time to start discussing it at the very least and make a proposal as a community goal based on the outcome of these discussions. чт, 6 окт. 2022 г., 07:48 Artem Goncharov <artem.goncharov@gmail.com>:
Hey,
If this is there mostly for audit purposes then I guess a more efficient solution is to introduce an audit table which will have no performance impact on the "current state" of service. Audit records are also never updated means performance here is relatively straight forward. Audit may become a "feature" with enable switch.
This should be a better solution rather then letting db permanently grow and forcing admins to constantly "fight" against it. This also gives a much cleaner audit experience. Actually this is also not a new approach and is being followed in many places (i.e. auditd)
Regards, Artem
---- typed from mobile, auto-correct typos assumed ----
On Thu, Oct 6, 2022, 07:27 Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Not having soft delete in the database is really quite bad for operators and it's not about tooling, but it's about audit purposes.
If take nova as example, this also means that once server is deleted, event log will be also wiped with no way to see who and when has performed delete action. And that is really used feature, as we got requests like "why my VM has disappeared" at very least one a week.
For other services having deleted_at at least points to the datetime where to search in the logs.
At the same time I don't see any issue in having soft delete. It's just a matter of one systemd-timer, and too concerned about performance can set it to 1 day, thus almost no impact on db performance.
So from operator perspective I can say this is very valuable feature and I personally do struggle regularly with neutron services where it's absent. And I would hate this to disappear at all, as it would be really a nightmare.
ср, 5 окт. 2022 г., 14:48 Stephen Finucane <stephenfin@redhat.com>:
👋
I'm planning on bringing this up in the nova rooms at the PTG in a few weeks, but I'm also raising it here since this potentially affects other service projects and I can't attend all of those room :)
Many projects use the concept of "soft delete" in their database models. A soft deletable model typically has two additional columns, 'deleted' and 'deleted_at'. When deleting such a model, instead of actually deleting the database row (i.e. 'DELETE FROM table WHERE condition'), we set 'deleted' to 'True' and populate the 'deleted_at' column. This is helpful for auditing purposes (e.g. you can inspect all resources ever created, even after they've been "deleted") but bad for database performance (your tables can grow without bound). To work around the performance issues, most projects implement some kind of archive or purge command that will allow operators to periodically clean up these deleted resources. However, at least in nova, we've long since come to the conclusion that soft deleting isn't as useful as initially suspected and the need to run these commands is additional work for no benefit. We've moved toward not using it for all new models.
With this said, it's going to be difficult to get away from soft-delete quickly. Not only are there database migrations involved, but operators will need to rework their tooling to adapt to a new, no-soft-delete world. As such, I'd like to propose a half-way measure of making soft-delete configurable. To do this, I'd like to add a new flag in oslo.db, '[database] enable_soft_delete'. When set to 'False' anyone using the 'SoftDeleteMixin' from oslo.db would see these models hard deleted rather than soft deleted when calling 'soft_delete'. This would avoid the need for operators to run the various project-specific purge tooling. The RFC patch for this is available for review [1]. I can also do this on a project-specific basis and have proposed a similar patch for nova [2], however, doing it in oslo.db means every project that uses 'SoftDeleteMixin' in their models will get this for free. Projects that don't (glance, cinder) can switch to using this mixin and also get it for free.
As noted above, I intend to discuss this in the nova room at the PTG, but I'd be interested in people's thoughts ahead of time. Do you think this is a good idea? Should we proceed with it? Perhaps there are there better ways to do this? Let me know!
Cheers, Stephen
[1] https://review.opendev.org/c/openstack/oslo.db/+/860407 [2] https://review.opendev.org/c/openstack/nova/+/860401
On 06/10, Dmitriy Rabotyagov wrote:
Oh, yes, this is a good alternative. Actually I was thinking about smth like "openstack <resource> event list" (as only nova does have that) for quite a while, but without having the resources to lead and help out in implementation across projects I didn't dare to raise this topic. But it's probably high time to start discussing it at the very least and make a proposal as a community goal based on the outcome of these discussions.
Hi, The Cinder team has also been exploring the idea of having transaction records to help operators see the series of operations on resources (to figure out what happened to a resource) as well as help see what operations are currently happening (great for planning upgrades or bouncing services to other nodes) [1]. If this is going to be a completely new feature in all projects we may want to agree on some commonalities such as naming and available functionality: - Transaction history, including deleted resources - Ongoing operations: - Detailed: All info for each of the transactions - Summary: - Global: i.e. 10 migrations, 3 attachments - By host: i.e. Host1: 5 migrations | Host2: 5 migrations, 3 attachments [1]: https://review.opendev.org/c/openstack/cinder-specs/+/845176/2/specs/zed/tra...
чт, 6 окт. 2022 г., 07:48 Artem Goncharov <artem.goncharov@gmail.com>:
Hey,
If this is there mostly for audit purposes then I guess a more efficient solution is to introduce an audit table which will have no performance impact on the "current state" of service. Audit records are also never updated means performance here is relatively straight forward. Audit may become a "feature" with enable switch.
This should be a better solution rather then letting db permanently grow and forcing admins to constantly "fight" against it. This also gives a much cleaner audit experience. Actually this is also not a new approach and is being followed in many places (i.e. auditd)
Regards, Artem
With this said, it's going to be difficult to get away from soft-delete quickly. Not only are there database migrations involved, but operators will need to rework their tooling to adapt to a new, no-soft-delete world. As such, I'd like to propose a half-way measure of making soft-delete configurable. To do this, I'd like to add a new flag in oslo.db, '[database] enable_soft_delete'. When set to 'False' anyone using the 'SoftDeleteMixin' from oslo.db would see these models hard deleted rather than soft deleted when calling 'soft_delete'. This would
I don't think it's a big deal, but from the Cinder perspective this would require additional work, because in our DB layer we only use the `soft_delete` method for 3 tables: Volume Types, Volume Type Access, and Group Type Access. All other tables use other mechanisms to do the soft deletes. Cheers, Gorka.
avoid the need for operators to run the various project-specific purge tooling. The RFC patch for this is available for review [1]. I can also do this on a project-specific basis and have proposed a similar patch for nova [2], however, doing it in oslo.db means every project that uses 'SoftDeleteMixin' in their models will get this for free. Projects that don't (glance, cinder) can switch to using this mixin and also get it for free.
As noted above, I intend to discuss this in the nova room at the PTG, but I'd be interested in people's thoughts ahead of time. Do you think this is a good idea? Should we proceed with it? Perhaps there are there better ways to do this? Let me know!
Cheers, Stephen
[1] https://review.opendev.org/c/openstack/oslo.db/+/860407 [2] https://review.opendev.org/c/openstack/nova/+/860401
Le jeu. 6 oct. 2022 à 07:54, Artem Goncharov <artem.goncharov@gmail.com> a écrit :
Hey,
If this is there mostly for audit purposes then I guess a more efficient solution is to introduce an audit table which will have no performance impact on the "current state" of service. Audit records are also never updated means performance here is relatively straight forward. Audit may become a "feature" with enable switch.
This should be a better solution rather then letting db permanently grow and forcing admins to constantly "fight" against it. This also gives a much cleaner audit experience. Actually this is also not a new approach and is being followed in many places (i.e. auditd)
This isn't true. Any operator that sees the Nova DBs [1] growing can use two commands (or just cron them) : nova-manage db archive_deleted_rows nova-manage db purge The first command will archive the soft-deleted records to another table and the second one will purge them. Here, I don't see why we would change this by a configuration option, but we can discuss this at the PTG like Stephen said. -Sylvain Regards,
Artem
---- typed from mobile, auto-correct typos assumed ----
On Thu, Oct 6, 2022, 07:27 Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Not having soft delete in the database is really quite bad for operators and it's not about tooling, but it's about audit purposes.
If take nova as example, this also means that once server is deleted, event log will be also wiped with no way to see who and when has performed delete action. And that is really used feature, as we got requests like "why my VM has disappeared" at very least one a week.
For other services having deleted_at at least points to the datetime where to search in the logs.
At the same time I don't see any issue in having soft delete. It's just a matter of one systemd-timer, and too concerned about performance can set it to 1 day, thus almost no impact on db performance.
So from operator perspective I can say this is very valuable feature and I personally do struggle regularly with neutron services where it's absent. And I would hate this to disappear at all, as it would be really a nightmare.
ср, 5 окт. 2022 г., 14:48 Stephen Finucane <stephenfin@redhat.com>:
👋
I'm planning on bringing this up in the nova rooms at the PTG in a few weeks, but I'm also raising it here since this potentially affects other service projects and I can't attend all of those room :)
Many projects use the concept of "soft delete" in their database models. A soft deletable model typically has two additional columns, 'deleted' and 'deleted_at'. When deleting such a model, instead of actually deleting the database row (i.e. 'DELETE FROM table WHERE condition'), we set 'deleted' to 'True' and populate the 'deleted_at' column. This is helpful for auditing purposes (e.g. you can inspect all resources ever created, even after they've been "deleted") but bad for database performance (your tables can grow without bound). To work around the performance issues, most projects implement some kind of archive or purge command that will allow operators to periodically clean up these deleted resources. However, at least in nova, we've long since come to the conclusion that soft deleting isn't as useful as initially suspected and the need to run these commands is additional work for no benefit. We've moved toward not using it for all new models.
With this said, it's going to be difficult to get away from soft-delete quickly. Not only are there database migrations involved, but operators will need to rework their tooling to adapt to a new, no-soft-delete world. As such, I'd like to propose a half-way measure of making soft-delete configurable. To do this, I'd like to add a new flag in oslo.db, '[database] enable_soft_delete'. When set to 'False' anyone using the 'SoftDeleteMixin' from oslo.db would see these models hard deleted rather than soft deleted when calling 'soft_delete'. This would avoid the need for operators to run the various project-specific purge tooling. The RFC patch for this is available for review [1]. I can also do this on a project-specific basis and have proposed a similar patch for nova [2], however, doing it in oslo.db means every project that uses 'SoftDeleteMixin' in their models will get this for free. Projects that don't (glance, cinder) can switch to using this mixin and also get it for free.
As noted above, I intend to discuss this in the nova room at the PTG, but I'd be interested in people's thoughts ahead of time. Do you think this is a good idea? Should we proceed with it? Perhaps there are there better ways to do this? Let me know!
Cheers, Stephen
[1] https://review.opendev.org/c/openstack/oslo.db/+/860407 [2] https://review.opendev.org/c/openstack/nova/+/860401
Le jeu. 6 oct. 2022 à 09:38, Sylvain Bauza <sbauza@redhat.com> a écrit :
Le jeu. 6 oct. 2022 à 07:54, Artem Goncharov <artem.goncharov@gmail.com> a écrit :
Hey,
If this is there mostly for audit purposes then I guess a more efficient solution is to introduce an audit table which will have no performance impact on the "current state" of service. Audit records are also never updated means performance here is relatively straight forward. Audit may become a "feature" with enable switch.
This should be a better solution rather then letting db permanently grow and forcing admins to constantly "fight" against it. This also gives a much cleaner audit experience. Actually this is also not a new approach and is being followed in many places (i.e. auditd)
This isn't true. Any operator that sees the Nova DBs [1] growing can use two commands (or just cron them) : nova-manage db archive_deleted_rows nova-manage db purge
The first command will archive the soft-deleted records to another table and the second one will purge them.
My apologies, forgot to add the link https://docs.openstack.org/nova/rocky/cli/nova-manage.html#nova-database Also, forgot the footnote [1] actually only the nova cell DBs are supporting soft-delete records, we don't have this for the API DB.
Here, I don't see why we would change this by a configuration option, but we can discuss this at the PTG like Stephen said. -Sylvain
Regards,
Artem
---- typed from mobile, auto-correct typos assumed ----
On Thu, Oct 6, 2022, 07:27 Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Not having soft delete in the database is really quite bad for operators and it's not about tooling, but it's about audit purposes.
If take nova as example, this also means that once server is deleted, event log will be also wiped with no way to see who and when has performed delete action. And that is really used feature, as we got requests like "why my VM has disappeared" at very least one a week.
For other services having deleted_at at least points to the datetime where to search in the logs.
At the same time I don't see any issue in having soft delete. It's just a matter of one systemd-timer, and too concerned about performance can set it to 1 day, thus almost no impact on db performance.
So from operator perspective I can say this is very valuable feature and I personally do struggle regularly with neutron services where it's absent. And I would hate this to disappear at all, as it would be really a nightmare.
ср, 5 окт. 2022 г., 14:48 Stephen Finucane <stephenfin@redhat.com>:
👋
I'm planning on bringing this up in the nova rooms at the PTG in a few weeks, but I'm also raising it here since this potentially affects other service projects and I can't attend all of those room :)
Many projects use the concept of "soft delete" in their database models. A soft deletable model typically has two additional columns, 'deleted' and 'deleted_at'. When deleting such a model, instead of actually deleting the database row (i.e. 'DELETE FROM table WHERE condition'), we set 'deleted' to 'True' and populate the 'deleted_at' column. This is helpful for auditing purposes (e.g. you can inspect all resources ever created, even after they've been "deleted") but bad for database performance (your tables can grow without bound). To work around the performance issues, most projects implement some kind of archive or purge command that will allow operators to periodically clean up these deleted resources. However, at least in nova, we've long since come to the conclusion that soft deleting isn't as useful as initially suspected and the need to run these commands is additional work for no benefit. We've moved toward not using it for all new models.
With this said, it's going to be difficult to get away from soft-delete quickly. Not only are there database migrations involved, but operators will need to rework their tooling to adapt to a new, no-soft-delete world. As such, I'd like to propose a half-way measure of making soft-delete configurable. To do this, I'd like to add a new flag in oslo.db, '[database] enable_soft_delete'. When set to 'False' anyone using the 'SoftDeleteMixin' from oslo.db would see these models hard deleted rather than soft deleted when calling 'soft_delete'. This would avoid the need for operators to run the various project-specific purge tooling. The RFC patch for this is available for review [1]. I can also do this on a project-specific basis and have proposed a similar patch for nova [2], however, doing it in oslo.db means every project that uses 'SoftDeleteMixin' in their models will get this for free. Projects that don't (glance, cinder) can switch to using this mixin and also get it for free.
As noted above, I intend to discuss this in the nova room at the PTG, but I'd be interested in people's thoughts ahead of time. Do you think this is a good idea? Should we proceed with it? Perhaps there are there better ways to do this? Let me know!
Cheers, Stephen
[1] https://review.opendev.org/c/openstack/oslo.db/+/860407 [2] https://review.opendev.org/c/openstack/nova/+/860401
Le jeu. 6 oct. 2022 à 09:48, Sylvain Bauza <sbauza@redhat.com> a écrit :
Le jeu. 6 oct. 2022 à 09:38, Sylvain Bauza <sbauza@redhat.com> a écrit :
Le jeu. 6 oct. 2022 à 07:54, Artem Goncharov <artem.goncharov@gmail.com> a écrit :
Hey,
If this is there mostly for audit purposes then I guess a more efficient solution is to introduce an audit table which will have no performance impact on the "current state" of service. Audit records are also never updated means performance here is relatively straight forward. Audit may become a "feature" with enable switch.
This should be a better solution rather then letting db permanently grow and forcing admins to constantly "fight" against it. This also gives a much cleaner audit experience. Actually this is also not a new approach and is being followed in many places (i.e. auditd)
This isn't true. Any operator that sees the Nova DBs [1] growing can use two commands (or just cron them) : nova-manage db archive_deleted_rows nova-manage db purge
The first command will archive the soft-deleted records to another table and the second one will purge them.
My apologies, forgot to add the link https://docs.openstack.org/nova/rocky/cli/nova-manage.html#nova-database
Morning not caffeinated yet, sorry. Wrong link : https://docs.openstack.org/nova/latest/cli/nova-manage.html#db-archive-delet... Also, forgot the footnote
[1] actually only the nova cell DBs are supporting soft-delete records, we don't have this for the API DB.
Here, I don't see why we would change this by a configuration option, but we can discuss this at the PTG like Stephen said. -Sylvain
Regards,
Artem
---- typed from mobile, auto-correct typos assumed ----
On Thu, Oct 6, 2022, 07:27 Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Not having soft delete in the database is really quite bad for operators and it's not about tooling, but it's about audit purposes.
If take nova as example, this also means that once server is deleted, event log will be also wiped with no way to see who and when has performed delete action. And that is really used feature, as we got requests like "why my VM has disappeared" at very least one a week.
For other services having deleted_at at least points to the datetime where to search in the logs.
At the same time I don't see any issue in having soft delete. It's just a matter of one systemd-timer, and too concerned about performance can set it to 1 day, thus almost no impact on db performance.
So from operator perspective I can say this is very valuable feature and I personally do struggle regularly with neutron services where it's absent. And I would hate this to disappear at all, as it would be really a nightmare.
ср, 5 окт. 2022 г., 14:48 Stephen Finucane <stephenfin@redhat.com>:
👋
I'm planning on bringing this up in the nova rooms at the PTG in a few weeks, but I'm also raising it here since this potentially affects other service projects and I can't attend all of those room :)
Many projects use the concept of "soft delete" in their database models. A soft deletable model typically has two additional columns, 'deleted' and 'deleted_at'. When deleting such a model, instead of actually deleting the database row (i.e. 'DELETE FROM table WHERE condition'), we set 'deleted' to 'True' and populate the 'deleted_at' column. This is helpful for auditing purposes (e.g. you can inspect all resources ever created, even after they've been "deleted") but bad for database performance (your tables can grow without bound). To work around the performance issues, most projects implement some kind of archive or purge command that will allow operators to periodically clean up these deleted resources. However, at least in nova, we've long since come to the conclusion that soft deleting isn't as useful as initially suspected and the need to run these commands is additional work for no benefit. We've moved toward not using it for all new models.
With this said, it's going to be difficult to get away from soft-delete quickly. Not only are there database migrations involved, but operators will need to rework their tooling to adapt to a new, no-soft-delete world. As such, I'd like to propose a half-way measure of making soft-delete configurable. To do this, I'd like to add a new flag in oslo.db, '[database] enable_soft_delete'. When set to 'False' anyone using the 'SoftDeleteMixin' from oslo.db would see these models hard deleted rather than soft deleted when calling 'soft_delete'. This would avoid the need for operators to run the various project-specific purge tooling. The RFC patch for this is available for review [1]. I can also do this on a project-specific basis and have proposed a similar patch for nova [2], however, doing it in oslo.db means every project that uses 'SoftDeleteMixin' in their models will get this for free. Projects that don't (glance, cinder) can switch to using this mixin and also get it for free.
As noted above, I intend to discuss this in the nova room at the PTG, but I'd be interested in people's thoughts ahead of time. Do you think this is a good idea? Should we proceed with it? Perhaps there are there better ways to do this? Let me know!
Cheers, Stephen
[1] https://review.opendev.org/c/openstack/oslo.db/+/860407 [2] https://review.opendev.org/c/openstack/nova/+/860401
Sylvain Bauza <sbauza@redhat.com> writes:
This isn't true. Any operator that sees the Nova DBs [1] growing can use two commands (or just cron them) : nova-manage db archive_deleted_rows nova-manage db purge
It's actually easier. Adding --purge to the first command removes the need to run the second. --Dan
I would like to strongly second this: not having soft delete, or an equivalent for audit purposes (I am not attached to the actual implementation), would be a great loss. We actually have a long standing task to add soft delete to Blazar, which I am hoping will be merged in Antelope. As an operator, I also get annoyed by the lack of soft delete in Neutron, for example to answer the question: who was using this specific floating IP at this specific time? On Thu, 6 Oct 2022 at 07:28, Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Not having soft delete in the database is really quite bad for operators and it's not about tooling, but it's about audit purposes.
If take nova as example, this also means that once server is deleted, event log will be also wiped with no way to see who and when has performed delete action. And that is really used feature, as we got requests like "why my VM has disappeared" at very least one a week.
For other services having deleted_at at least points to the datetime where to search in the logs.
At the same time I don't see any issue in having soft delete. It's just a matter of one systemd-timer, and too concerned about performance can set it to 1 day, thus almost no impact on db performance.
So from operator perspective I can say this is very valuable feature and I personally do struggle regularly with neutron services where it's absent. And I would hate this to disappear at all, as it would be really a nightmare.
ср, 5 окт. 2022 г., 14:48 Stephen Finucane <stephenfin@redhat.com>:
👋
I'm planning on bringing this up in the nova rooms at the PTG in a few weeks, but I'm also raising it here since this potentially affects other service projects and I can't attend all of those room :)
Many projects use the concept of "soft delete" in their database models. A soft deletable model typically has two additional columns, 'deleted' and 'deleted_at'. When deleting such a model, instead of actually deleting the database row (i.e. 'DELETE FROM table WHERE condition'), we set 'deleted' to 'True' and populate the 'deleted_at' column. This is helpful for auditing purposes (e.g. you can inspect all resources ever created, even after they've been "deleted") but bad for database performance (your tables can grow without bound). To work around the performance issues, most projects implement some kind of archive or purge command that will allow operators to periodically clean up these deleted resources. However, at least in nova, we've long since come to the conclusion that soft deleting isn't as useful as initially suspected and the need to run these commands is additional work for no benefit. We've moved toward not using it for all new models.
With this said, it's going to be difficult to get away from soft-delete quickly. Not only are there database migrations involved, but operators will need to rework their tooling to adapt to a new, no-soft-delete world. As such, I'd like to propose a half-way measure of making soft-delete configurable. To do this, I'd like to add a new flag in oslo.db, '[database] enable_soft_delete'. When set to 'False' anyone using the 'SoftDeleteMixin' from oslo.db would see these models hard deleted rather than soft deleted when calling 'soft_delete'. This would avoid the need for operators to run the various project-specific purge tooling. The RFC patch for this is available for review [1]. I can also do this on a project-specific basis and have proposed a similar patch for nova [2], however, doing it in oslo.db means every project that uses 'SoftDeleteMixin' in their models will get this for free. Projects that don't (glance, cinder) can switch to using this mixin and also get it for free.
As noted above, I intend to discuss this in the nova room at the PTG, but I'd be interested in people's thoughts ahead of time. Do you think this is a good idea? Should we proceed with it? Perhaps there are there better ways to do this? Let me know!
Cheers, Stephen
[1] https://review.opendev.org/c/openstack/oslo.db/+/860407 [2] https://review.opendev.org/c/openstack/nova/+/860401
participants (7)
-
Artem Goncharov
-
Dan Smith
-
Dmitriy Rabotyagov
-
Gorka Eguileor
-
Pierre Riteau
-
Stephen Finucane
-
Sylvain Bauza