[largescale-sig] OpenStack DB Archiver
Hello large-scalers! TLDR: we opensource a tool to help reducing size of databases. See https://github.com/ovh/osarchiver/ Few months ago, we released a tool, name osarchiver, which we are using on our production environment (at OVH) to help reduce the size of our tables in mariadb (or mysql) In fact, some tables are well know to grow very quickly. We use it, for example, to clean the OpenStack mistral database from old tasks, actions and executions which are older than a year. Another use case could be to archive some data in another table (e.g. with _archived as suffix) if they are 6 months old, and delete this data after 1 year. The source code of this tool is available here: https://github.com/ovh/osarchiver/ We were wondering if some other users would be interested in using the tool, and maybe move it under the opendev governance? Feel free to contact us and/or answer this thread. Cheers, -- Arnaud, Pierre-Samuel and OVH team
On 7/16/20 3:31 PM, Arnaud Morin wrote:
Hello large-scalers!
TLDR: we opensource a tool to help reducing size of databases. See https://github.com/ovh/osarchiver/
Few months ago, we released a tool, name osarchiver, which we are using on our production environment (at OVH) to help reduce the size of our tables in mariadb (or mysql)
In fact, some tables are well know to grow very quickly.
We use it, for example, to clean the OpenStack mistral database from old tasks, actions and executions which are older than a year.
Another use case could be to archive some data in another table (e.g. with _archived as suffix) if they are 6 months old, and delete this data after 1 year.
The source code of this tool is available here: https://github.com/ovh/osarchiver/
We were wondering if some other users would be interested in using the tool, and maybe move it under the opendev governance?
Feel free to contact us and/or answer this thread.
Cheers,
Hi, That's very nice, thanks a lot for releasing such a thing. However, there's room for improvement if you would like to see your tool shipped everywhere: - please define a requirements.txt - please get the debian folder away from the main master branch, especially considering it's using dh_virtualenv !!! - please tag with a release number Also, with what release of OpenStack has this been tested? Is this bound to a specific release? Cheers, Thomas Goirand (zigo)
Thomas Goirand <thomas@goirand.fr> wrote on ven. [2020-juil.-17 09:47:22 +0200]:
On 7/16/20 3:31 PM, Arnaud Morin wrote:
Hello large-scalers!
TLDR: we opensource a tool to help reducing size of databases. See https://github.com/ovh/osarchiver/
Few months ago, we released a tool, name osarchiver, which we are using on our production environment (at OVH) to help reduce the size of our tables in mariadb (or mysql)
In fact, some tables are well know to grow very quickly.
We use it, for example, to clean the OpenStack mistral database from old tasks, actions and executions which are older than a year.
Another use case could be to archive some data in another table (e.g. with _archived as suffix) if they are 6 months old, and delete this data after 1 year.
The source code of this tool is available here: https://github.com/ovh/osarchiver/
We were wondering if some other users would be interested in using the tool, and maybe move it under the opendev governance?
Feel free to contact us and/or answer this thread.
Cheers,
Hi,
That's very nice, thanks a lot for releasing such a thing.
However, there's room for improvement if you would like to see your tool shipped everywhere:
- please define a requirements.txt - please get the debian folder away from the main master branch, especially considering it's using dh_virtualenv !!! - please tag with a release number
Also, with what release of OpenStack has this been tested? Is this bound to a specific release?
Cheers,
Thomas Goirand (zigo)
Hi Thomas, Thanks for your answer. We will update the repository accordingly. We tested OSArchiver on Newton and Stein releases of OpenStack. By design the tool is agnostic and rely only on the presence of the 'deleted_at' column so for now we do not expect to be bound to a specific release. Best regards, -- PS
Arnaud Morin wrote:
[...] The source code of this tool is available here: https://github.com/ovh/osarchiver/
Thanks for sharing this tool!
We were wondering if some other users would be interested in using the tool, and maybe move it under the opendev governance?
I think this is one of those small operational tools that everyone ends up reinventing in their corner, duplicating effort. I support the idea of pushing it upstream, as it will make it easier for others to improve it, but also make the tool more discoverable. In terms of governance, we have several paths we could follow: 1/ we could create a specific project team to maintain this. That sounds completely overkill given the scope and size of the tool, and the fact that it's mostly feature-complete. Project teams are great to produce new "openstack" service components, but this is more peripheral operational tooling that would be released independently. 2/ we could adopt it at the Large Scale SIG, and promote it from there. I feel like this is useful beyond Large scale deployments though, so that sounds suboptimal 3/ during the last Opendev event we discussed reviving the OSops[1] idea: a lightweight area where operators can share the various small tools that they end up creating to help them operate OpenStack deployments. The effort has been dormant for a few years. I personally think the last option is the best, even if we need to figure a few things out before we can land this. I'll start a separate thread on OSops, and depending on how that goes, we'll choose between option 3 or option 2 for osarchiver. [1] https://wiki.openstack.org/wiki/Osops -- Thierry Carrez (ttx)
Hi everyone, During the last Opendev event we discussed reviving the OSops[1] idea: a lightweight area where operators can share the various small tools that they end up creating to help them operate OpenStack deployments. The effort has been mostly dormant for a few years. We had a recent thread[2] about osarchiver, a new operators helper, and whether it would make sense to push it upstream. I think the best option would be to revive OSops and land it there. Who is interested in helping to revive/maintain this ? If we revive it, I think we should move its repositories away from the catch-all "x" directory under opendev, which was created for projects that were not claimed by anyone during the big migration. If Osops should be considered distinct from OpenStack, then I'd recommend giving it its own opendev top directory, and move existing x/osops-* repositories to osops/*. If we'd like to make OSops a product of the OpenStack community (and have contributions to it be fully recognized as contributions to "OpenStack"), then I'd recommend creating a specific SIG dedicated to this, and move the x/osops-* repositories to openstack/osops-*. [1] https://wiki.openstack.org/wiki/Osops [2] http://lists.openstack.org/pipermail/openstack-discuss/2020-July/015977.html -- Thierry Carrez (ttx)
+1 for reviving OSops. Within OVH, we choose OpenStack for its community and openness, so we will try to participate in such effort! Cheers, -- Arnaud Morin On 17.07.20 - 15:19, Thierry Carrez wrote:
Hi everyone,
During the last Opendev event we discussed reviving the OSops[1] idea: a lightweight area where operators can share the various small tools that they end up creating to help them operate OpenStack deployments. The effort has been mostly dormant for a few years.
We had a recent thread[2] about osarchiver, a new operators helper, and whether it would make sense to push it upstream. I think the best option would be to revive OSops and land it there.
Who is interested in helping to revive/maintain this ?
If we revive it, I think we should move its repositories away from the catch-all "x" directory under opendev, which was created for projects that were not claimed by anyone during the big migration.
If Osops should be considered distinct from OpenStack, then I'd recommend giving it its own opendev top directory, and move existing x/osops-* repositories to osops/*.
If we'd like to make OSops a product of the OpenStack community (and have contributions to it be fully recognized as contributions to "OpenStack"), then I'd recommend creating a specific SIG dedicated to this, and move the x/osops-* repositories to openstack/osops-*.
[1] https://wiki.openstack.org/wiki/Osops [2] http://lists.openstack.org/pipermail/openstack-discuss/2020-July/015977.html
-- Thierry Carrez (ttx)
On 7/17/20 8:19 AM, Thierry Carrez wrote:
If Osops should be considered distinct from OpenStack
That feels like the wrong statement to make, even if only implicitly by repo organization. Is there a compelling reason not to have osops under the openstack namespace?
If Osops should be considered distinct from OpenStack
That feels like the wrong statement to make, even if only implicitly by repo organization. Is there a compelling reason not to have osops under the openstack namespace?
I think it makes the most sense to be under the openstack namespace. We have the Operations Docs SIG right now that took on some of the operator-specific documentation that no longer had a home. This was a consistent issue brought up in the Ops Meetup events. While not "wildly successful" in getting a bunch of new and updated docs, it at least has accomplished the main goal of getting these docs published to docs.openstack.org again, and providing a place where more collaboration can (and occasionally does) happen to improve those docs. I think we could probably expand the scope of this SIG. Especially considering it is a pretty low-volume SIG anyway. I would be good with changing this to something like the "Operator Docs and Tooling SIG" and getting any of these useful tooling repos under governance through that. I personally wouldn't be able to spend a lot of time working on anything under the SIG, but I'd be happy to keep an eye out for any new reviews and help get those through. Sean
+1 on combining this in with the existing SiG and efforts. Amy (spotz) On Mon, Jul 27, 2020 at 1:02 PM Sean McGinnis <sean.mcginnis@gmx.com> wrote:
If Osops should be considered distinct from OpenStack
That feels like the wrong statement to make, even if only implicitly by repo organization. Is there a compelling reason not to have osops under the openstack namespace?
I think it makes the most sense to be under the openstack namespace.
We have the Operations Docs SIG right now that took on some of the operator-specific documentation that no longer had a home. This was a consistent issue brought up in the Ops Meetup events. While not "wildly successful" in getting a bunch of new and updated docs, it at least has accomplished the main goal of getting these docs published to docs.openstack.org again, and providing a place where more collaboration can (and occasionally does) happen to improve those docs.
I think we could probably expand the scope of this SIG. Especially considering it is a pretty low-volume SIG anyway. I would be good with changing this to something like the "Operator Docs and Tooling SIG" and getting any of these useful tooling repos under governance through that. I personally wouldn't be able to spend a lot of time working on anything under the SIG, but I'd be happy to keep an eye out for any new reviews and help get those through.
Sean
Interested in this as well. We use Openstack a $Dayjob :) On Mon, Jul 27, 2020 at 2:52 PM Amy Marrich <amy@demarco.com> wrote:
+1 on combining this in with the existing SiG and efforts.
Amy (spotz)
On Mon, Jul 27, 2020 at 1:02 PM Sean McGinnis <sean.mcginnis@gmx.com> wrote:
If Osops should be considered distinct from OpenStack
That feels like the wrong statement to make, even if only implicitly by repo organization. Is there a compelling reason not to have osops under the openstack namespace?
I think it makes the most sense to be under the openstack namespace.
We have the Operations Docs SIG right now that took on some of the operator-specific documentation that no longer had a home. This was a consistent issue brought up in the Ops Meetup events. While not "wildly successful" in getting a bunch of new and updated docs, it at least has accomplished the main goal of getting these docs published to docs.openstack.org again, and providing a place where more collaboration can (and occasionally does) happen to improve those docs.
I think we could probably expand the scope of this SIG. Especially considering it is a pretty low-volume SIG anyway. I would be good with changing this to something like the "Operator Docs and Tooling SIG" and getting any of these useful tooling repos under governance through that. I personally wouldn't be able to spend a lot of time working on anything under the SIG, but I'd be happy to keep an eye out for any new reviews and help get those through.
Sean
+1 Laurent Dumont <laurentfdumont@gmail.com> schrieb am Mi., 29. Juli 2020, 04:00:
Interested in this as well. We use Openstack a $Dayjob :)
On Mon, Jul 27, 2020 at 2:52 PM Amy Marrich <amy@demarco.com> wrote:
+1 on combining this in with the existing SiG and efforts.
Amy (spotz)
On Mon, Jul 27, 2020 at 1:02 PM Sean McGinnis <sean.mcginnis@gmx.com> wrote:
If Osops should be considered distinct from OpenStack
That feels like the wrong statement to make, even if only implicitly by repo organization. Is there a compelling reason not to have osops under the openstack namespace?
I think it makes the most sense to be under the openstack namespace.
We have the Operations Docs SIG right now that took on some of the operator-specific documentation that no longer had a home. This was a consistent issue brought up in the Ops Meetup events. While not "wildly successful" in getting a bunch of new and updated docs, it at least has accomplished the main goal of getting these docs published to docs.openstack.org again, and providing a place where more collaboration can (and occasionally does) happen to improve those docs.
I think we could probably expand the scope of this SIG. Especially considering it is a pretty low-volume SIG anyway. I would be good with changing this to something like the "Operator Docs and Tooling SIG" and getting any of these useful tooling repos under governance through that. I personally wouldn't be able to spend a lot of time working on anything under the SIG, but I'd be happy to keep an eye out for any new reviews and help get those through.
Sean
+1 to put these in the Operations Docs SIG On Wed, Jul 29, 2020 at 12:25 AM Fabian Zimmermann <dev.faz@gmail.com> wrote:
+1
Laurent Dumont <laurentfdumont@gmail.com> schrieb am Mi., 29. Juli 2020, 04:00:
Interested in this as well. We use Openstack a $Dayjob :)
On Mon, Jul 27, 2020 at 2:52 PM Amy Marrich <amy@demarco.com> wrote:
+1 on combining this in with the existing SiG and efforts.
Amy (spotz)
On Mon, Jul 27, 2020 at 1:02 PM Sean McGinnis <sean.mcginnis@gmx.com> wrote:
If Osops should be considered distinct from OpenStack
That feels like the wrong statement to make, even if only implicitly by repo organization. Is there a compelling reason not to have osops under the openstack namespace?
I think it makes the most sense to be under the openstack namespace.
We have the Operations Docs SIG right now that took on some of the operator-specific documentation that no longer had a home. This was a consistent issue brought up in the Ops Meetup events. While not "wildly successful" in getting a bunch of new and updated docs, it at least has accomplished the main goal of getting these docs published to docs.openstack.org again, and providing a place where more collaboration can (and occasionally does) happen to improve those docs.
I think we could probably expand the scope of this SIG. Especially considering it is a pretty low-volume SIG anyway. I would be good with changing this to something like the "Operator Docs and Tooling SIG" and getting any of these useful tooling repos under governance through that. I personally wouldn't be able to spend a lot of time working on anything under the SIG, but I'd be happy to keep an eye out for any new reviews and help get those through.
Sean
-- Chris Morgan <mihalis68@gmail.com>
I have proposed https://review.opendev.org/#/c/744005/ to expand the scope of the Operations Docs SIG to include tooling like this. Sean On 7/30/20 7:52 AM, Chris Morgan wrote:
+1 to put these in the Operations Docs SIG
On Wed, Jul 29, 2020 at 12:25 AM Fabian Zimmermann <dev.faz@gmail.com <mailto:dev.faz@gmail.com>> wrote:
+1
Laurent Dumont <laurentfdumont@gmail.com <mailto:laurentfdumont@gmail.com>> schrieb am Mi., 29. Juli 2020, 04:00:
Interested in this as well. We use Openstack a $Dayjob :)
On Mon, Jul 27, 2020 at 2:52 PM Amy Marrich <amy@demarco.com <mailto:amy@demarco.com>> wrote:
+1 on combining this in with the existing SiG and efforts.
Amy (spotz)
On Mon, Jul 27, 2020 at 1:02 PM Sean McGinnis <sean.mcginnis@gmx.com <mailto:sean.mcginnis@gmx.com>> wrote:
>> If Osops should be considered distinct from OpenStack > > That feels like the wrong statement to make, even if only implicitly > by repo organization. Is there a compelling reason not to have osops > under the openstack namespace? > I think it makes the most sense to be under the openstack namespace.
We have the Operations Docs SIG right now that took on some of the operator-specific documentation that no longer had a home. This was a consistent issue brought up in the Ops Meetup events. While not "wildly successful" in getting a bunch of new and updated docs, it at least has accomplished the main goal of getting these docs published to docs.openstack.org <http://docs.openstack.org> again, and providing a place where more collaboration can (and occasionally does) happen to improve those docs.
I think we could probably expand the scope of this SIG. Especially considering it is a pretty low-volume SIG anyway. I would be good with changing this to something like the "Operator Docs and Tooling SIG" and getting any of these useful tooling repos under governance through that. I personally wouldn't be able to spend a lot of time working on anything under the SIG, but I'd be happy to keep an eye out for any new reviews and help get those through.
Sean
-- Chris Morgan <mihalis68@gmail.com <mailto:mihalis68@gmail.com>>
Arnaud, thanks for sharing this tool. On Thu, Jul 16, 2020 at 3:48 PM Arnaud Morin <arnaud.morin@gmail.com> wrote:
Hello large-scalers!
TLDR: we opensource a tool to help reducing size of databases. See https://github.com/ovh/osarchiver/
Few months ago, we released a tool, name osarchiver, which we are using on our production environment (at OVH) to help reduce the size of our tables in mariadb (or mysql)
In fact, some tables are well know to grow very quickly.
We use it, for example, to clean the OpenStack mistral database from old tasks, actions and executions which are older than a year.
Another use case could be to archive some data in another table (e.g. with _archived as suffix) if they are 6 months old, and delete this data after 1 year.
The source code of this tool is available here: https://github.com/ovh/osarchiver/
We were wondering if some other users would be interested in using the tool, and maybe move it under the opendev governance?
Feel free to contact us and/or answer this thread.
Cheers,
-- Arnaud, Pierre-Samuel and OVH team
Arnaud Morin wrote:
[...] We were wondering if some other users would be interested in using the tool, and maybe move it under the opendev governance?
Resurrecting this thread, as OSops has now been revived under the auspices of the OpenStack Operation Docs and Tooling SIG. There are basically 3 potential ways forward for OSarchiver: 1- Keep it as-is on GitHub, and reference it where we can in OpenStack docs 2- Relicense it under Apache-2 and move it in a subdirectory under openstack/osops 3- Move it under its own repository under opendev and propose it as a new official OpenStack project (relicensing under Apache-2 will be necessary if accepted) Options (1) and (3) have the benefit of keeping it under its own repository. Options (2) and (3) have the benefit of counting towards an official OpenStack contribution. Options (1) and (2) have the benefit of not requiring TC approval. All other things being equal, if the end goal is to increase discoverability, option 3 is probably the best. Regards, -- Thierry Carrez
On Fri, 2021-01-29 at 13:47 +0100, Thierry Carrez wrote:
Arnaud Morin wrote:
[...] We were wondering if some other users would be interested in using the tool, and maybe move it under the opendev governance?
Resurrecting this thread, as OSops has now been revived under the auspices of the OpenStack Operation Docs and Tooling SIG.
There are basically 3 potential ways forward for OSarchiver:
1- Keep it as-is on GitHub, and reference it where we can in OpenStack docs
2- Relicense it under Apache-2 and move it in a subdirectory under openstack/osops
3- Move it under its own repository under opendev and propose it as a new official OpenStack project (relicensing under Apache-2 will be necessary if accepted)
Options (1) and (3) have the benefit of keeping it under its own repository. Options (2) and (3) have the benefit of counting towards an official OpenStack contribution. Options (1) and (2) have the benefit of not requiring TC approval.
All other things being equal, if the end goal is to increase discoverability, option 3 is probably the best.
not to detract form the converation on where to host it, but now that i have discoverd this via this thread i have one quetion. OSarchiver appears to be bypassing the shadow tabels whcih the project maintian to allow you to archive rows in the the project db in a different table. instad OSarchiver chooese to archive it in an external DB or file we have talked about wheter or not we can remove shaddow tables in nova enteirly a few times in the past but we did not want to break operators that actully use them but it appares OVH at least has developed there own alrenitive presumably becasue the proejcts own archive and purge functionality was not meetingyour need. would the option to disable shadow tabels or define a retention policy for delete rows be useful to operators or is this even a capablity that project coudl declare out of scope and delegate that to a new openstack porject e.g. opeiton 3 above to do instead? im not sure how suportable OSarchiver would be in our downstream product right now but with testing it might be somethign we could look at includign in the futrue. we currently rely on chron jobs to invoke nova-magne ecta to achive similar functionality to OSarchiver but if that chron job breaks its hard to detect and the delete rows can build up causing really slow db queries. as a seperate service with loggin i assuem this is simplere to monitor and alarm on if it fails since it provices one central point to manage the archival and deletion of rows so i kind of like this approch even if its direct db access right now would make it unsupportable in our product without veting the code and productising the repo via ooo integration.
Regards,
Hello, At first, we designed OSarchiver to be openstack agnostic. It can be executed on any kind of database, the only thing it needs is a column in the DB to read in order to perform the removal. We are using it on our cloud, executing it in a cron just like we would do nova-manage. We monitor the output and raise an alert if something is broken. If I remember correctly we used it first on our mistral DB, I dont know if mistral implement this "shadow" table mechanism that you describe for nova, so that's maybe why we initiate this project at first. Moreover, I dont know the details about how nova move data into shadow tables, but were more confident in re-using our OSarchiver on nova after it had prove a good work on mistral :) On our side, we would agree with something like a retention policy of data, but I dont believe all the projects can/would implement such functionality, so the approach to have an external tool doing it is IMHO a path we can follow. About the 3 options for OSarchiver that Thierry proposed, we dont have a strong opinion but options 2 or 3 OK for us. Option 3 would be the best, but we dont know what is the necesary work that should be done on the code itself before this option become viable. Finally, no problem to move under APACHE2 license. Cheers, -- Arnaud Morin On 29.01.21 - 14:22, Sean Mooney wrote:
On Fri, 2021-01-29 at 13:47 +0100, Thierry Carrez wrote:
Arnaud Morin wrote:
[...] We were wondering if some other users would be interested in using the tool, and maybe move it under the opendev governance?
Resurrecting this thread, as OSops has now been revived under the auspices of the OpenStack Operation Docs and Tooling SIG.
There are basically 3 potential ways forward for OSarchiver:
1- Keep it as-is on GitHub, and reference it where we can in OpenStack docs
2- Relicense it under Apache-2 and move it in a subdirectory under openstack/osops
3- Move it under its own repository under opendev and propose it as a new official OpenStack project (relicensing under Apache-2 will be necessary if accepted)
Options (1) and (3) have the benefit of keeping it under its own repository. Options (2) and (3) have the benefit of counting towards an official OpenStack contribution. Options (1) and (2) have the benefit of not requiring TC approval.
All other things being equal, if the end goal is to increase discoverability, option 3 is probably the best.
not to detract form the converation on where to host it, but now that i have discoverd this via this thread i have one quetion. OSarchiver appears to be bypassing the shadow tabels whcih the project maintian to allow you to archive rows in the the project db in a different table.
instad OSarchiver chooese to archive it in an external DB or file
we have talked about wheter or not we can remove shaddow tables in nova enteirly a few times in the past but we did not want to break operators that actully use them but it appares OVH at least has developed there own alrenitive presumably becasue the proejcts own archive and purge functionality was not meetingyour need.
would the option to disable shadow tabels or define a retention policy for delete rows be useful to operators or is this even a capablity that project coudl declare out of scope and delegate that to a new openstack porject e.g. opeiton 3 above to do instead?
im not sure how suportable OSarchiver would be in our downstream product right now but with testing it might be somethign we could look at includign in the futrue. we currently rely on chron jobs to invoke nova-magne ecta to achive similar functionality to OSarchiver but if that chron job breaks its hard to detect and the delete rows can build up causing really slow db queries. as a seperate service with loggin i assuem this is simplere to monitor and alarm on if it fails since it provices one central point to manage the archival and deletion of rows so i kind of like this approch even if its direct db access right now would make it unsupportable in our product without veting the code and productising the repo via ooo integration.
Regards,
Hi, at CERN, Nova shadow tables are used to keep the deleted entries for 30 days. We have a cronjob, that runs every day, to move the deleted rows into the shadow tables and purge them using nova-manage (deleted rows are kept in the shadow tables for 30 days). For other projects that keep deleted rows but don't have shadow tables, for example Glance, we purge the tables everyday with glance-manage (again, keeping the deleted rows for 30 days). Belmiro On Fri, Jan 29, 2021 at 3:28 PM Sean Mooney <smooney@redhat.com> wrote:
On Fri, 2021-01-29 at 13:47 +0100, Thierry Carrez wrote:
Arnaud Morin wrote:
[...] We were wondering if some other users would be interested in using the tool, and maybe move it under the opendev governance?
Resurrecting this thread, as OSops has now been revived under the auspices of the OpenStack Operation Docs and Tooling SIG.
There are basically 3 potential ways forward for OSarchiver:
1- Keep it as-is on GitHub, and reference it where we can in OpenStack docs
2- Relicense it under Apache-2 and move it in a subdirectory under openstack/osops
3- Move it under its own repository under opendev and propose it as a new official OpenStack project (relicensing under Apache-2 will be necessary if accepted)
Options (1) and (3) have the benefit of keeping it under its own repository. Options (2) and (3) have the benefit of counting towards an official OpenStack contribution. Options (1) and (2) have the benefit of not requiring TC approval.
All other things being equal, if the end goal is to increase discoverability, option 3 is probably the best.
not to detract form the converation on where to host it, but now that i have discoverd this via this thread i have one quetion. OSarchiver appears to be bypassing the shadow tabels whcih the project maintian to allow you to archive rows in the the project db in a different table.
instad OSarchiver chooese to archive it in an external DB or file
we have talked about wheter or not we can remove shaddow tables in nova enteirly a few times in the past but we did not want to break operators that actully use them but it appares OVH at least has developed there own alrenitive presumably becasue the proejcts own archive and purge functionality was not meetingyour need.
would the option to disable shadow tabels or define a retention policy for delete rows be useful to operators or is this even a capablity that project coudl declare out of scope and delegate that to a new openstack porject e.g. opeiton 3 above to do instead?
im not sure how suportable OSarchiver would be in our downstream product right now but with testing it might be somethign we could look at includign in the futrue. we currently rely on chron jobs to invoke nova-magne ecta to achive similar functionality to OSarchiver but if that chron job breaks its hard to detect and the delete rows can build up causing really slow db queries. as a seperate service with loggin i assuem this is simplere to monitor and alarm on if it fails since it provices one central point to manage the archival and deletion of rows so i kind of like this approch even if its direct db access right now would make it unsupportable in our product without veting the code and productising the repo via ooo integration.
Regards,
On Wed, 2021-02-10 at 12:37 +0100, Belmiro Moreira wrote:
Hi, at CERN, Nova shadow tables are used to keep the deleted entries for 30 days. We have a cronjob, that runs every day, to move the deleted rows into the shadow tables and purge them using nova-manage (deleted rows are kept in the shadow tables for 30 days).
For other projects that keep deleted rows but don't have shadow tables, for example Glance, we purge the tables everyday with glance-manage (again, keeping the deleted rows for 30 days).
thanks that is useful. that is what i was expecting to be a good defualt setup keep for 30 days but run daily with no row limit. that way you are archiving/deleteing 1 days worth of entries every day but keeping them for 30 days before they are removed.
Belmiro
On Fri, Jan 29, 2021 at 3:28 PM Sean Mooney <smooney@redhat.com> wrote:
On Fri, 2021-01-29 at 13:47 +0100, Thierry Carrez wrote:
Arnaud Morin wrote:
[...] We were wondering if some other users would be interested in using the tool, and maybe move it under the opendev governance?
Resurrecting this thread, as OSops has now been revived under the auspices of the OpenStack Operation Docs and Tooling SIG.
There are basically 3 potential ways forward for OSarchiver:
1- Keep it as-is on GitHub, and reference it where we can in OpenStack docs
2- Relicense it under Apache-2 and move it in a subdirectory under openstack/osops
3- Move it under its own repository under opendev and propose it as a new official OpenStack project (relicensing under Apache-2 will be necessary if accepted)
Options (1) and (3) have the benefit of keeping it under its own repository. Options (2) and (3) have the benefit of counting towards an official OpenStack contribution. Options (1) and (2) have the benefit of not requiring TC approval.
All other things being equal, if the end goal is to increase discoverability, option 3 is probably the best.
not to detract form the converation on where to host it, but now that i have discoverd this via this thread i have one quetion. OSarchiver appears to be bypassing the shadow tabels whcih the project maintian to allow you to archive rows in the the project db in a different table.
instad OSarchiver chooese to archive it in an external DB or file
we have talked about wheter or not we can remove shaddow tables in nova enteirly a few times in the past but we did not want to break operators that actully use them but it appares OVH at least has developed there own alrenitive presumably becasue the proejcts own archive and purge functionality was not meetingyour need.
would the option to disable shadow tabels or define a retention policy for delete rows be useful to operators or is this even a capablity that project coudl declare out of scope and delegate that to a new openstack porject e.g. opeiton 3 above to do instead?
im not sure how suportable OSarchiver would be in our downstream product right now but with testing it might be somethign we could look at includign in the futrue. we currently rely on chron jobs to invoke nova-magne ecta to achive similar functionality to OSarchiver but if that chron job breaks its hard to detect and the delete rows can build up causing really slow db queries. as a seperate service with loggin i assuem this is simplere to monitor and alarm on if it fails since it provices one central point to manage the archival and deletion of rows so i kind of like this approch even if its direct db access right now would make it unsupportable in our product without veting the code and productising the repo via ooo integration.
Regards,
participants (12)
-
Amy Marrich
-
Arnaud Morin
-
Belmiro Moreira
-
Ben Nemec
-
Chris Morgan
-
Fabian Zimmermann
-
Laurent Dumont
-
Pierre-Samuel LE STANG
-
Sean McGinnis
-
Sean Mooney
-
Thierry Carrez
-
Thomas Goirand