[ops] database archiving tool

Belmiro Moreira moreira.belmiro.email.lists at gmail.com
Thu May 9 15:43:49 UTC 2019

Hi Pierre-Samuel,
at this point most of the OpenStack projects have their own way to
archive/delete soft deleted records.
But one thing usually missing is the retention period of soft deleted
records and then the archived data.

I'm interested to learn more about what you are doing.
Is there any link to access the code?


On Thu, May 9, 2019 at 5:25 PM Pierre-Samuel LE STANG <
pierre-samuel.le-stang at corp.ovh.com> wrote:

> Hi all,
> At OVH we needed to write our own tool that archive data from OpenStack
> databases to prevent some side effect related to huge tables (slower
> response
> time, changing MariaDB query plan) and to answer to some legal aspects.
> So we started to write a python tool which is called OSArchiver that I
> briefly
> presented at Denver few days ago in the "Optimizing OpenStack at large
> scale"
> talk. We think that this tool could be helpful to other and are ready to
> open
> source it, first we would like to get the opinion of the ops community
> about
> that tool.
> To sum-up OSArchiver is written to work regardless of Openstack project.
> The
> tool relies on the fact that soft deleted data are recognizable because of
> their 'deleted' column which is set to 1 or uuid and 'deleted_at' column
> which
> is set to the date of deletion.
> The points to have in mind about OSArchiver:
> * There is no knowledge of business objects
> * One table might be archived if it contains 'deleted' column
> * Children rows are archived before parents rows
> * A row can not be deleted if it fails to be archived
> Here are features already implemented:
> * Archive data in an other database and/or file (actually SQL and CSV
> formats are supported) to be easily imported
> * Delete data from Openstack databases
> * Customizable (retention, exclude DBs, exclude tables, bulk insert/delete)
> * Multiple archiving configuration
> * Dry-run mode
> * Easily extensible, you can add your own destination module (other file
> format, remote storage etc...)
> * Archive and/or delete only mode
> It also means that by design you can run osarchiver not only on OpenStack
> databases but also on archived OpenStack databases.
> Thanks in advance for your feedbacks.
> --
> Pierre-Samuel Le Stang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190509/6a150aef/attachment-0001.html>

More information about the openstack-discuss mailing list