Open Stack

Le jeu. 6 oct. 2022 à 09:38, Sylvain Bauza <sbauza at redhat.com> a écrit :

>
>
> Le jeu. 6 oct. 2022 à 07:54, Artem Goncharov <artem.goncharov at gmail.com>
> a écrit :
>
>> Hey,
>>
>> If this is there mostly for audit purposes then I guess a more efficient
>> solution is to introduce an audit table which will have no performance
>> impact on the "current state" of service. Audit records are also never
>> updated means performance here is relatively straight forward. Audit may
>> become a "feature" with enable switch.
>>
>> This should be a better solution rather then letting db permanently grow
>> and forcing admins to constantly "fight" against it. This also gives a much
>> cleaner audit experience. Actually this is also not a new approach and is
>> being followed in many places (i.e. auditd)
>>
>>
> This isn't true. Any operator that sees the Nova DBs [1] growing can use
> two commands (or just cron them) :
> nova-manage db archive_deleted_rows
> nova-manage db purge
>
> The first command will archive the soft-deleted records to another table
> and the second one will purge them.
>

My apologies, forgot to add the link
https://docs.openstack.org/nova/rocky/cli/nova-manage.html#nova-database

Also, forgot the footnote
[1] actually only the nova cell DBs are supporting soft-delete records, we
don't have this for the API DB.


> Here, I don't see why we would change this by a configuration option, but
> we can discuss this at the PTG like Stephen said.
> -Sylvain
>
>
> Regards,
>> Artem
>>
>> ----
>> typed from mobile, auto-correct typos assumed
>> ----
>>
>> On Thu, Oct 6, 2022, 07:27 Dmitriy Rabotyagov <noonedeadpunk at gmail.com>
>> wrote:
>>
>>> Not having soft delete in the database is really quite bad for operators
>>> and it's not about tooling, but it's about audit purposes.
>>>
>>> If take nova as example, this also means that once server is deleted,
>>> event log will be also wiped with no way to see who and when has performed
>>> delete action. And that is really used feature, as we got requests like
>>> "why my VM has disappeared" at very least one a week.
>>>
>>> For other services having deleted_at at least points to the datetime
>>> where to search in the logs.
>>>
>>> At the same time I don't see any issue in having soft delete. It's just
>>> a matter of one systemd-timer, and too concerned about performance can set
>>> it to 1 day, thus almost no impact on db performance.
>>>
>>> So from operator perspective I can say this is very valuable feature and
>>> I personally do struggle regularly with neutron services where it's absent.
>>> And I would hate this to disappear at all, as it would be really a
>>> nightmare.
>>>
>>> ср, 5 окт. 2022 г., 14:48 Stephen Finucane <stephenfin at redhat.com>:
>>>
>>>> 👋
>>>>
>>>> I'm planning on bringing this up in the nova rooms at the PTG in a few
>>>> weeks,
>>>> but I'm also raising it here since this potentially affects other
>>>> service
>>>> projects and I can't attend all of those room :)
>>>>
>>>> Many projects use the concept of "soft delete" in their database
>>>> models. A soft
>>>> deletable model typically has two additional columns, 'deleted' and
>>>> 'deleted_at'. When deleting such a model, instead of actually deleting
>>>> the
>>>> database row (i.e. 'DELETE FROM table WHERE condition'), we set
>>>> 'deleted' to
>>>> 'True' and populate the 'deleted_at' column. This is helpful for
>>>> auditing
>>>> purposes (e.g. you can inspect all resources ever created, even after
>>>> they've
>>>> been "deleted") but bad for database performance (your tables can grow
>>>> without
>>>> bound). To work around the performance issues, most projects implement
>>>> some kind
>>>> of archive or purge command that will allow operators to periodically
>>>> clean up
>>>> these deleted resources. However, at least in nova, we've long since
>>>> come to the
>>>> conclusion that soft deleting isn't as useful as initially suspected
>>>> and the
>>>> need to run these commands is additional work for no benefit. We've
>>>> moved toward
>>>> not using it for all new models.
>>>>
>>>> With this said, it's going to be difficult to get away from soft-delete
>>>> quickly.
>>>> Not only are there database migrations involved, but operators will
>>>> need to
>>>> rework their tooling to adapt to a new, no-soft-delete world. As such,
>>>> I'd like
>>>> to propose a half-way measure of making soft-delete configurable. To do
>>>> this,
>>>> I'd like to add a new flag in oslo.db, '[database] enable_soft_delete'.
>>>> When set
>>>> to 'False' anyone using the 'SoftDeleteMixin' from oslo.db would see
>>>> these
>>>> models hard deleted rather than soft deleted when calling
>>>> 'soft_delete'. This
>>>> would avoid the need for operators to run the various project-specific
>>>> purge
>>>> tooling. The RFC patch for this is available for review [1]. I can also
>>>> do this
>>>> on a project-specific basis and have proposed a similar patch for nova
>>>> [2],
>>>> however, doing it in oslo.db means every project that uses
>>>> 'SoftDeleteMixin' in
>>>> their models will get this for free. Projects that don't (glance,
>>>> cinder) can
>>>> switch to using this mixin and also get it for free.
>>>>
>>>> As noted above, I intend to discuss this in the nova room at the PTG,
>>>> but I'd be
>>>> interested in people's thoughts ahead of time. Do you think this is a
>>>> good idea?
>>>> Should we proceed with it? Perhaps there are there better ways to do
>>>> this? Let
>>>> me know!
>>>>
>>>> Cheers,
>>>> Stephen
>>>>
>>>> [1] https://review.opendev.org/c/openstack/oslo.db/+/860407
>>>> [2] https://review.opendev.org/c/openstack/nova/+/860401
>>>>
>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20221006/00bc74ef/attachment.htm>