Open Stack

Le jeu. 6 oct. 2022 à 07:54, Artem Goncharov <artem.goncharov at gmail.com> a
écrit :

> Hey,
>
> If this is there mostly for audit purposes then I guess a more efficient
> solution is to introduce an audit table which will have no performance
> impact on the "current state" of service. Audit records are also never
> updated means performance here is relatively straight forward. Audit may
> become a "feature" with enable switch.
>
> This should be a better solution rather then letting db permanently grow
> and forcing admins to constantly "fight" against it. This also gives a much
> cleaner audit experience. Actually this is also not a new approach and is
> being followed in many places (i.e. auditd)
>
>
This isn't true. Any operator that sees the Nova DBs [1] growing can use
two commands (or just cron them) :
nova-manage db archive_deleted_rows
nova-manage db purge

The first command will archive the soft-deleted records to another table
and the second one will purge them.

Here, I don't see why we would change this by a configuration option, but
we can discuss this at the PTG like Stephen said.
-Sylvain


Regards,
> Artem
>
> ----
> typed from mobile, auto-correct typos assumed
> ----
>
> On Thu, Oct 6, 2022, 07:27 Dmitriy Rabotyagov <noonedeadpunk at gmail.com>
> wrote:
>
>> Not having soft delete in the database is really quite bad for operators
>> and it's not about tooling, but it's about audit purposes.
>>
>> If take nova as example, this also means that once server is deleted,
>> event log will be also wiped with no way to see who and when has performed
>> delete action. And that is really used feature, as we got requests like
>> "why my VM has disappeared" at very least one a week.
>>
>> For other services having deleted_at at least points to the datetime
>> where to search in the logs.
>>
>> At the same time I don't see any issue in having soft delete. It's just a
>> matter of one systemd-timer, and too concerned about performance can set it
>> to 1 day, thus almost no impact on db performance.
>>
>> So from operator perspective I can say this is very valuable feature and
>> I personally do struggle regularly with neutron services where it's absent.
>> And I would hate this to disappear at all, as it would be really a
>> nightmare.
>>
>> ср, 5 окт. 2022 г., 14:48 Stephen Finucane <stephenfin at redhat.com>:
>>
>>> 👋
>>>
>>> I'm planning on bringing this up in the nova rooms at the PTG in a few
>>> weeks,
>>> but I'm also raising it here since this potentially affects other service
>>> projects and I can't attend all of those room :)
>>>
>>> Many projects use the concept of "soft delete" in their database models.
>>> A soft
>>> deletable model typically has two additional columns, 'deleted' and
>>> 'deleted_at'. When deleting such a model, instead of actually deleting
>>> the
>>> database row (i.e. 'DELETE FROM table WHERE condition'), we set
>>> 'deleted' to
>>> 'True' and populate the 'deleted_at' column. This is helpful for auditing
>>> purposes (e.g. you can inspect all resources ever created, even after
>>> they've
>>> been "deleted") but bad for database performance (your tables can grow
>>> without
>>> bound). To work around the performance issues, most projects implement
>>> some kind
>>> of archive or purge command that will allow operators to periodically
>>> clean up
>>> these deleted resources. However, at least in nova, we've long since
>>> come to the
>>> conclusion that soft deleting isn't as useful as initially suspected and
>>> the
>>> need to run these commands is additional work for no benefit. We've
>>> moved toward
>>> not using it for all new models.
>>>
>>> With this said, it's going to be difficult to get away from soft-delete
>>> quickly.
>>> Not only are there database migrations involved, but operators will need
>>> to
>>> rework their tooling to adapt to a new, no-soft-delete world. As such,
>>> I'd like
>>> to propose a half-way measure of making soft-delete configurable. To do
>>> this,
>>> I'd like to add a new flag in oslo.db, '[database] enable_soft_delete'.
>>> When set
>>> to 'False' anyone using the 'SoftDeleteMixin' from oslo.db would see
>>> these
>>> models hard deleted rather than soft deleted when calling 'soft_delete'.
>>> This
>>> would avoid the need for operators to run the various project-specific
>>> purge
>>> tooling. The RFC patch for this is available for review [1]. I can also
>>> do this
>>> on a project-specific basis and have proposed a similar patch for nova
>>> [2],
>>> however, doing it in oslo.db means every project that uses
>>> 'SoftDeleteMixin' in
>>> their models will get this for free. Projects that don't (glance,
>>> cinder) can
>>> switch to using this mixin and also get it for free.
>>>
>>> As noted above, I intend to discuss this in the nova room at the PTG,
>>> but I'd be
>>> interested in people's thoughts ahead of time. Do you think this is a
>>> good idea?
>>> Should we proceed with it? Perhaps there are there better ways to do
>>> this? Let
>>> me know!
>>>
>>> Cheers,
>>> Stephen
>>>
>>> [1] https://review.opendev.org/c/openstack/oslo.db/+/860407
>>> [2] https://review.opendev.org/c/openstack/nova/+/860401
>>>
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20221006/a022ac9a/attachment-0001.htm>