Open Stack

Le jeu. 6 oct. 2022 à 09:48, Sylvain Bauza <sbauza at redhat.com> a écrit :

>
>
> Le jeu. 6 oct. 2022 à 09:38, Sylvain Bauza <sbauza at redhat.com> a écrit :
>
>>
>>
>> Le jeu. 6 oct. 2022 à 07:54, Artem Goncharov <artem.goncharov at gmail.com>
>> a écrit :
>>
>>> Hey,
>>>
>>> If this is there mostly for audit purposes then I guess a more efficient
>>> solution is to introduce an audit table which will have no performance
>>> impact on the "current state" of service. Audit records are also never
>>> updated means performance here is relatively straight forward. Audit may
>>> become a "feature" with enable switch.
>>>
>>> This should be a better solution rather then letting db permanently grow
>>> and forcing admins to constantly "fight" against it. This also gives a much
>>> cleaner audit experience. Actually this is also not a new approach and is
>>> being followed in many places (i.e. auditd)
>>>
>>>
>> This isn't true. Any operator that sees the Nova DBs [1] growing can use
>> two commands (or just cron them) :
>> nova-manage db archive_deleted_rows
>> nova-manage db purge
>>
>> The first command will archive the soft-deleted records to another table
>> and the second one will purge them.
>>
>
> My apologies, forgot to add the link
> https://docs.openstack.org/nova/rocky/cli/nova-manage.html#nova-database
>
>
Morning not caffeinated yet, sorry.
Wrong link :
https://docs.openstack.org/nova/latest/cli/nova-manage.html#db-archive-deleted-rows

Also, forgot the footnote
> [1] actually only the nova cell DBs are supporting soft-delete records, we
> don't have this for the API DB.
>
>
>> Here, I don't see why we would change this by a configuration option, but
>> we can discuss this at the PTG like Stephen said.
>> -Sylvain
>>
>>
>> Regards,
>>> Artem
>>>
>>> ----
>>> typed from mobile, auto-correct typos assumed
>>> ----
>>>
>>> On Thu, Oct 6, 2022, 07:27 Dmitriy Rabotyagov <noonedeadpunk at gmail.com>
>>> wrote:
>>>
>>>> Not having soft delete in the database is really quite bad for
>>>> operators and it's not about tooling, but it's about audit purposes.
>>>>
>>>> If take nova as example, this also means that once server is deleted,
>>>> event log will be also wiped with no way to see who and when has performed
>>>> delete action. And that is really used feature, as we got requests like
>>>> "why my VM has disappeared" at very least one a week.
>>>>
>>>> For other services having deleted_at at least points to the datetime
>>>> where to search in the logs.
>>>>
>>>> At the same time I don't see any issue in having soft delete. It's just
>>>> a matter of one systemd-timer, and too concerned about performance can set
>>>> it to 1 day, thus almost no impact on db performance.
>>>>
>>>> So from operator perspective I can say this is very valuable feature
>>>> and I personally do struggle regularly with neutron services where it's
>>>> absent. And I would hate this to disappear at all, as it would be really a
>>>> nightmare.
>>>>
>>>> ср, 5 окт. 2022 г., 14:48 Stephen Finucane <stephenfin at redhat.com>:
>>>>
>>>>> 👋
>>>>>
>>>>> I'm planning on bringing this up in the nova rooms at the PTG in a few
>>>>> weeks,
>>>>> but I'm also raising it here since this potentially affects other
>>>>> service
>>>>> projects and I can't attend all of those room :)
>>>>>
>>>>> Many projects use the concept of "soft delete" in their database
>>>>> models. A soft
>>>>> deletable model typically has two additional columns, 'deleted' and
>>>>> 'deleted_at'. When deleting such a model, instead of actually deleting
>>>>> the
>>>>> database row (i.e. 'DELETE FROM table WHERE condition'), we set
>>>>> 'deleted' to
>>>>> 'True' and populate the 'deleted_at' column. This is helpful for
>>>>> auditing
>>>>> purposes (e.g. you can inspect all resources ever created, even after
>>>>> they've
>>>>> been "deleted") but bad for database performance (your tables can grow
>>>>> without
>>>>> bound). To work around the performance issues, most projects implement
>>>>> some kind
>>>>> of archive or purge command that will allow operators to periodically
>>>>> clean up
>>>>> these deleted resources. However, at least in nova, we've long since
>>>>> come to the
>>>>> conclusion that soft deleting isn't as useful as initially suspected
>>>>> and the
>>>>> need to run these commands is additional work for no benefit. We've
>>>>> moved toward
>>>>> not using it for all new models.
>>>>>
>>>>> With this said, it's going to be difficult to get away from
>>>>> soft-delete quickly.
>>>>> Not only are there database migrations involved, but operators will
>>>>> need to
>>>>> rework their tooling to adapt to a new, no-soft-delete world. As such,
>>>>> I'd like
>>>>> to propose a half-way measure of making soft-delete configurable. To
>>>>> do this,
>>>>> I'd like to add a new flag in oslo.db, '[database]
>>>>> enable_soft_delete'. When set
>>>>> to 'False' anyone using the 'SoftDeleteMixin' from oslo.db would see
>>>>> these
>>>>> models hard deleted rather than soft deleted when calling
>>>>> 'soft_delete'. This
>>>>> would avoid the need for operators to run the various project-specific
>>>>> purge
>>>>> tooling. The RFC patch for this is available for review [1]. I can
>>>>> also do this
>>>>> on a project-specific basis and have proposed a similar patch for nova
>>>>> [2],
>>>>> however, doing it in oslo.db means every project that uses
>>>>> 'SoftDeleteMixin' in
>>>>> their models will get this for free. Projects that don't (glance,
>>>>> cinder) can
>>>>> switch to using this mixin and also get it for free.
>>>>>
>>>>> As noted above, I intend to discuss this in the nova room at the PTG,
>>>>> but I'd be
>>>>> interested in people's thoughts ahead of time. Do you think this is a
>>>>> good idea?
>>>>> Should we proceed with it? Perhaps there are there better ways to do
>>>>> this? Let
>>>>> me know!
>>>>>
>>>>> Cheers,
>>>>> Stephen
>>>>>
>>>>> [1] https://review.opendev.org/c/openstack/oslo.db/+/860407
>>>>> [2] https://review.opendev.org/c/openstack/nova/+/860401
>>>>>
>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20221006/987d750f/attachment-0001.htm>