Open Stack

Hey,

If this is there mostly for audit purposes then I guess a more efficient
solution is to introduce an audit table which will have no performance
impact on the "current state" of service. Audit records are also never
updated means performance here is relatively straight forward. Audit may
become a "feature" with enable switch.

This should be a better solution rather then letting db permanently grow
and forcing admins to constantly "fight" against it. This also gives a much
cleaner audit experience. Actually this is also not a new approach and is
being followed in many places (i.e. auditd)

Regards,
Artem

----
typed from mobile, auto-correct typos assumed
----

On Thu, Oct 6, 2022, 07:27 Dmitriy Rabotyagov <noonedeadpunk at gmail.com>
wrote:

> Not having soft delete in the database is really quite bad for operators
> and it's not about tooling, but it's about audit purposes.
>
> If take nova as example, this also means that once server is deleted,
> event log will be also wiped with no way to see who and when has performed
> delete action. And that is really used feature, as we got requests like
> "why my VM has disappeared" at very least one a week.
>
> For other services having deleted_at at least points to the datetime where
> to search in the logs.
>
> At the same time I don't see any issue in having soft delete. It's just a
> matter of one systemd-timer, and too concerned about performance can set it
> to 1 day, thus almost no impact on db performance.
>
> So from operator perspective I can say this is very valuable feature and I
> personally do struggle regularly with neutron services where it's absent.
> And I would hate this to disappear at all, as it would be really a
> nightmare.
>
> ср, 5 окт. 2022 г., 14:48 Stephen Finucane <stephenfin at redhat.com>:
>
>> 👋
>>
>> I'm planning on bringing this up in the nova rooms at the PTG in a few
>> weeks,
>> but I'm also raising it here since this potentially affects other service
>> projects and I can't attend all of those room :)
>>
>> Many projects use the concept of "soft delete" in their database models.
>> A soft
>> deletable model typically has two additional columns, 'deleted' and
>> 'deleted_at'. When deleting such a model, instead of actually deleting the
>> database row (i.e. 'DELETE FROM table WHERE condition'), we set 'deleted'
>> to
>> 'True' and populate the 'deleted_at' column. This is helpful for auditing
>> purposes (e.g. you can inspect all resources ever created, even after
>> they've
>> been "deleted") but bad for database performance (your tables can grow
>> without
>> bound). To work around the performance issues, most projects implement
>> some kind
>> of archive or purge command that will allow operators to periodically
>> clean up
>> these deleted resources. However, at least in nova, we've long since come
>> to the
>> conclusion that soft deleting isn't as useful as initially suspected and
>> the
>> need to run these commands is additional work for no benefit. We've moved
>> toward
>> not using it for all new models.
>>
>> With this said, it's going to be difficult to get away from soft-delete
>> quickly.
>> Not only are there database migrations involved, but operators will need
>> to
>> rework their tooling to adapt to a new, no-soft-delete world. As such,
>> I'd like
>> to propose a half-way measure of making soft-delete configurable. To do
>> this,
>> I'd like to add a new flag in oslo.db, '[database] enable_soft_delete'.
>> When set
>> to 'False' anyone using the 'SoftDeleteMixin' from oslo.db would see these
>> models hard deleted rather than soft deleted when calling 'soft_delete'.
>> This
>> would avoid the need for operators to run the various project-specific
>> purge
>> tooling. The RFC patch for this is available for review [1]. I can also
>> do this
>> on a project-specific basis and have proposed a similar patch for nova
>> [2],
>> however, doing it in oslo.db means every project that uses
>> 'SoftDeleteMixin' in
>> their models will get this for free. Projects that don't (glance, cinder)
>> can
>> switch to using this mixin and also get it for free.
>>
>> As noted above, I intend to discuss this in the nova room at the PTG, but
>> I'd be
>> interested in people's thoughts ahead of time. Do you think this is a
>> good idea?
>> Should we proceed with it? Perhaps there are there better ways to do
>> this? Let
>> me know!
>>
>> Cheers,
>> Stephen
>>
>> [1] https://review.opendev.org/c/openstack/oslo.db/+/860407
>> [2] https://review.opendev.org/c/openstack/nova/+/860401
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20221006/5444d3c3/attachment-0001.htm>