Open Stack

Hello Belmiro,

I will put the code on OVH's github repository as soon as possible. I'll keep
you informed.

-- 
Pierre-Samuel Le Stang

Belmiro Moreira <moreira.belmiro.email.lists at gmail.com> wrote on jeu. [2019-mai-09 17:43:49 +0200]:
> Hi Pierre-Samuel,
> at this point most of the OpenStack projects have their own way to archive/
> delete soft deleted records.
> But one thing usually missing is the retention period of soft deleted records
> and then the archived data.
> 
> I'm interested to learn more about what you are doing.
> Is there any link to access the code?
> 
> Belmiro
> CERN
> 
> On Thu, May 9, 2019 at 5:25 PM Pierre-Samuel LE STANG <
> pierre-samuel.le-stang at corp.ovh.com> wrote:
> 
>     Hi all,
> 
>     At OVH we needed to write our own tool that archive data from OpenStack
>     databases to prevent some side effect related to huge tables (slower
>     response
>     time, changing MariaDB query plan) and to answer to some legal aspects.
> 
>     So we started to write a python tool which is called OSArchiver that I
>     briefly
>     presented at Denver few days ago in the "Optimizing OpenStack at large
>     scale"
>     talk. We think that this tool could be helpful to other and are ready to
>     open
>     source it, first we would like to get the opinion of the ops community
>     about
>     that tool.
> 
>     To sum-up OSArchiver is written to work regardless of Openstack project.
>     The
>     tool relies on the fact that soft deleted data are recognizable because of
>     their 'deleted' column which is set to 1 or uuid and 'deleted_at' column
>     which
>     is set to the date of deletion.
> 
>     The points to have in mind about OSArchiver:
>     * There is no knowledge of business objects
>     * One table might be archived if it contains 'deleted' column
>     * Children rows are archived before parents rows
>     * A row can not be deleted if it fails to be archived
> 
>     Here are features already implemented:
>     * Archive data in an other database and/or file (actually SQL and CSV
>     formats are supported) to be easily imported
>     * Delete data from Openstack databases
>     * Customizable (retention, exclude DBs, exclude tables, bulk insert/delete)
>     * Multiple archiving configuration
>     * Dry-run mode
>     * Easily extensible, you can add your own destination module (other file
>     format, remote storage etc...)
>     * Archive and/or delete only mode
> 
>     It also means that by design you can run osarchiver not only on OpenStack
>     databases but also on archived OpenStack databases.
> 
>     Thanks in advance for your feedbacks.
> 
>     --
>     Pierre-Samuel Le Stang
> 
> 

-- 
Pierre-Samuel Le Stang