[largescale-sig] OpenStack DB Archiver

Arnaud Morin arnaud.morin at gmail.com
Fri Feb 5 08:32:34 UTC 2021


Hello,

At first, we designed OSarchiver to be openstack agnostic. It can be
executed on any kind of database, the only thing it needs is a column in
the DB to read in order to perform the removal.

We are using it on our cloud, executing it in a cron just like we would
do nova-manage. We monitor the output and raise an alert if something is
broken.

If I remember correctly we used it first on our mistral DB, I dont know
if mistral implement this "shadow" table mechanism that you describe for
nova, so that's maybe why we initiate this project at first.

Moreover, I dont know the details about how nova move data into shadow
tables, but were more confident in re-using our OSarchiver on nova after
it had prove a good work on mistral :)

On our side, we would agree with something like a retention policy of
data, but I dont believe all the projects can/would implement such
functionality, so the approach to have an external tool doing it is IMHO
a path we can follow.

About the 3 options for OSarchiver that Thierry proposed, we dont have a
strong opinion but options 2 or 3 OK for us.
Option 3 would be the best, but we dont know what is the necesary work
that should be done on the code itself before this option become viable.
Finally, no problem to move under APACHE2 license.

Cheers,


-- 
Arnaud Morin

On 29.01.21 - 14:22, Sean Mooney wrote:
> On Fri, 2021-01-29 at 13:47 +0100, Thierry Carrez wrote:
> > Arnaud Morin wrote:
> > > [...]
> > > We were wondering if some other users would be interested in using the
> > > tool, and maybe move it under the opendev governance?
> > 
> > Resurrecting this thread, as OSops has now been revived under the 
> > auspices of the OpenStack Operation Docs and Tooling SIG.
> > 
> > There are basically 3 potential ways forward for OSarchiver:
> > 
> > 1- Keep it as-is on GitHub, and reference it where we can in OpenStack docs
> > 
> > 2- Relicense it under Apache-2 and move it in a subdirectory under 
> > openstack/osops
> > 
> > 3- Move it under its own repository under opendev and propose it as a 
> > new official OpenStack project (relicensing under Apache-2 will be 
> > necessary if accepted)
> > 
> > Options (1) and (3) have the benefit of keeping it under its own 
> > repository. Options (2) and (3) have the benefit of counting towards an 
> > official OpenStack contribution. Options (1) and (2) have the benefit of 
> > not requiring TC approval.
> > 
> > All other things being equal, if the end goal is to increase 
> > discoverability, option 3 is probably the best.
> 
> not to detract form the converation on where to host it, but now that i have discoverd this
> via this thread i have one quetion. OSarchiver appears to be bypassing the shadow tabels
> whcih the project maintian to allow you to archive rows in the the project db in a different table.
> 
> instad OSarchiver chooese to archive it in an external DB or file
> 
> we have talked about wheter or not we can remove shaddow tables in nova enteirly a few times in the past
> but we did not want to break operators that actully use them but it appares OVH at least has developed there
> own alrenitive presumably becasue the proejcts own archive and purge functionality was not meetingyour need.
> 
> would the option to disable shadow tabels or define a retention policy for delete rows be useful to operators
> or is this even a capablity that project coudl declare out of scope and delegate that to a new openstack porject
> e.g. opeiton 3 above to do instead?
> 
> im not sure how suportable OSarchiver would be in our downstream product right now but with testing it might be
> somethign we could look at includign in the futrue. we currently rely on chron jobs to invoke nova-magne ecta
> to achive similar functionality to OSarchiver but if that chron job breaks its hard to detect and the delete rows
> can build up causing really slow db queries.  as a seperate service with loggin i assuem this is simplere to monitor
> and alarm on if it fails since it provices one central point to manage the archival and deletion of rows so i kind
> of like this approch even if its direct db access right now would make it unsupportable in our product without veting
> the code and productising the repo via ooo integration.
> 
> 
> 
> > 
> > Regards,
> > 
> 
> 
> 



More information about the openstack-discuss mailing list