Ospurge or "project purge" - What's the right approach to cleanup projects prior to deletion
Hello Openstack-Discuss,
I was wondering what the right approach to clean up all resources of a project was in order for it to be deleted?
1) There is ospurge (https://opendev.org/x/ospurge) which works, but seems kind of abandoned looking at the last commits and pending issues (https://bugs.launchpad.net/ospurge). Will there still be any further development of this tool?
2) Then there is the project purge command (https://docs.openstack.org/python-openstackclient/victoria/cli/command-objec...) introduced with the Pike release (https://docs.openstack.org/releasenotes/python-openstackclient/pike.html#rel... // https://bugs.launchpad.net/python-openstackclient/+bug/1584596 // https://review.opendev.org/c/openstack/python-openstackclient/+/367673/).
While it seems only natural to have the cleanup / purge functionality built into the openstack client, I was wondering .. .
a) if all potentially existing resources types are covered and this can fully replace ospurge to allow any project to be cleaned up and deleted? What about i.e. Networks, Containers or objects?
b) Also ospurge seems to still have some extended options to target or exclude certain resources by using the following options:
[--resource {Backups,Snapshots,Volumes,Zones,Images,Stacks,FloatingIPs,RouterInterfaces,Routers,Ports,Networks,SecurityGroups,Servers,LoadBalancers,Receivers,Policies,Clusters,Profiles,Objects,Containers}] [--exclude-resource {Backups,Snapshots,Volumes,Zones,Images,Stacks,FloatingIPs,RouterInterfaces,Routers,Ports,Networks,SecurityGroups,Servers,LoadBalancers,Receivers,Policies,Clusters,Profiles,Objects,Containers}]
those options are not (yet) available for the "project purge" command of the openstack client.
With regards,
Christian
Hi,
Please check https://review.opendev.org/c/openstack/python-openstackclient/+/734485 https://review.opendev.org/c/openstack/python-openstackclient/+/734485 out. This is a followup of the ospurge, which bases on the project cleanup functionality built into OpenStackSDK. Idea is to replace “openstack project purge” functionality to completely rely on SDK only. Please also leave comments whether this works for you or not.
Regards, Artem
On 18. Jan 2021, at 17:24, Christian Rohmann christian.rohmann@inovex.de wrote:
Hello Openstack-Discuss,
I was wondering what the right approach to clean up all resources of a project was in order for it to be deleted?
- There is ospurge (https://opendev.org/x/ospurge) which works, but seems kind of abandoned looking at the last commits and pending issues (https://bugs.launchpad.net/ospurge).
Will there still be any further development of this tool?
- Then there is the project purge command (https://docs.openstack.org/python-openstackclient/victoria/cli/command-objec...) introduced with the
Pike release (https://docs.openstack.org/releasenotes/python-openstackclient/pike.html#rel... // https://bugs.launchpad.net/python-openstackclient/+bug/1584596 // https://review.opendev.org/c/openstack/python-openstackclient/+/367673/).
While it seems only natural to have the cleanup / purge functionality built into the openstack client, I was wondering .. .
a) if all potentially existing resources types are covered and this can fully replace ospurge to allow any project to be cleaned up and deleted? What about i.e. Networks, Containers or objects?
b) Also ospurge seems to still have some extended options to target or exclude certain resources by using the following options:
[--resource {Backups,Snapshots,Volumes,Zones,Images,Stacks,FloatingIPs,RouterInterfaces,Routers,Ports,Networks,SecurityGroups,Servers,LoadBalancers,Receivers,Policies,Clusters,Profiles,Objects,Containers}] [--exclude-resource {Backups,Snapshots,Volumes,Zones,Images,Stacks,FloatingIPs,RouterInterfaces,Routers,Ports,Networks,SecurityGroups,Servers,LoadBalancers,Receivers,Policies,Clusters,Profiles,Objects,Containers}]
those options are not (yet) available for the "project purge" command of the openstack client.
With regards,
Christian
On 1/18/21 6:10 PM, Artem Goncharov wrote:
Hi,
Please check https://review.opendev.org/c/openstack/python-openstackclient/+/734485 https://review.opendev.org/c/openstack/python-openstackclient/+/734485 out. This is a followup of the ospurge, which bases on the project cleanup functionality built into OpenStackSDK. Idea is to replace “openstack project purge” functionality to completely rely on SDK only. Please also leave comments whether this works for you or not.
Regards, Artem
Hi,
While all of this looks like a good idea, it's not implementing anything at all, just a few command line options. So it's currently of no use for operators.
I'm still saluting the initiative, but in 6 months, it's kind of slow moving, IMO.
Cheers,
Thomas Goirand (zigo)
What do you mean it doesn’t implement anything at all? It does clean up compute, network, block_storage, orchestrate resources. Moreover it gives you possibility to clean “old” resources (created before or last updated before).
On 18. Jan 2021, at 18:30, Thomas Goirand zigo@debian.org wrote:
On 1/18/21 6:10 PM, Artem Goncharov wrote:
Hi,
Please check https://review.opendev.org/c/openstack/python-openstackclient/+/734485 https://review.opendev.org/c/openstack/python-openstackclient/+/734485 out. This is a followup of the ospurge, which bases on the project cleanup functionality built into OpenStackSDK. Idea is to replace “openstack project purge” functionality to completely rely on SDK only. Please also leave comments whether this works for you or not.
Regards, Artem
Hi,
While all of this looks like a good idea, it's not implementing anything at all, just a few command line options. So it's currently of no use for operators.
I'm still saluting the initiative, but in 6 months, it's kind of slow moving, IMO.
Cheers,
Thomas Goirand (zigo)
On 1/18/21 6:56 PM, Artem Goncharov wrote:
What do you mean it doesn’t implement anything at all? It does clean up compute, network, block_storage, orchestrate resources. Moreover it gives you possibility to clean “old” resources (created before or last updated before).
Oh really? With that few lines of code? I'll re-read the patch then, sorry for my bad assumptions.
Can you point at the part that's actually deleting the resources?
Thomas Goirand (zigo)
Ha, thats exactly the case, the whole logic sits in sdk and is spread across the supported services: - https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/compu... https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/compute/v2/_proxy.py#L1798 - for compute. KeyPairs not dropped, since they belong to user, and not to the “project”; - https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/block... https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/block_storage/v3/_proxy.py#L547 - block storage; - https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/orche... https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/orchestration/v1/_proxy.py#L490 - https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/netwo... https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/network/v2/_proxy.py#L4130 - the most complex one in order to give possibility to clean “old” resource without destroying everything else
Adding image is few lines of code (never had enough time to add it), identity is a bit tricky, since also here mostly resources does not belong to Project. DNS would be also easy to do. OSC here is only providing I/F, while the logic sits in SDK and can be very easy extended for other services.
P.S. I use it this on an hourly basis since more than a year already (not a complete cleanup, but with update_after filter in project where the cloud is monitored).
Regards, Artem
On 18. Jan 2021, at 19:52, Thomas Goirand zigo@debian.org wrote:
On 1/18/21 6:56 PM, Artem Goncharov wrote:
What do you mean it doesn’t implement anything at all? It does clean up compute, network, block_storage, orchestrate resources. Moreover it gives you possibility to clean “old” resources (created before or last updated before).
Oh really? With that few lines of code? I'll re-read the patch then, sorry for my bad assumptions.
Can you point at the part that's actually deleting the resources?
Thomas Goirand (zigo)
Hey Artem,
thank you very much for your quick reply and pointer to the patchset you work on!
On 18/01/2021 20:14, Artem Goncharov wrote:
Ha, thats exactly the case, the whole logic sits in sdk and is spread across the supported services:
https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/compu... https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/compute/v2/_proxy.py#L1798 - for compute. KeyPairs not dropped, since they belong to user, and not to the “project”;
https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/block... https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/block_storage/v3/_proxy.py#L547 - block storage;
https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/orche... https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/orchestration/v1/_proxy.py#L490
https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/netwo... https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/network/v2/_proxy.py#L4130 - the most complex one in order to give possibility to clean “old” resource without destroying everything else
Adding image is few lines of code (never had enough time to add it), identity is a bit tricky, since also here mostly resources does not belong to Project. DNS would be also easy to do. OSC here is only providing I/F, while the logic sits in SDK and can be very easy extended for other services.
On 18. Jan 2021, at 19:52, Thomas Goirand <zigo@debian.org mailto:zigo@debian.org> wrote:
On 1/18/21 6:56 PM, Artem Goncharov wrote:
What do you mean it doesn’t implement anything at all? It does clean up compute, network, block_storage, orchestrate resources. Moreover it gives you possibility to clean “old” resources (created before or last updated before).
Oh really? With that few lines of code? I'll re-read the patch then, sorry for my bad assumptions.
Can you point at the part that's actually deleting the resources?
If I understood correctly, the cleanup relies on the SDK functionality / requirement for each resource type to provide a corresponding function( https://github.com/openstack/openstacksdk/blob/master/openstack/cloud/openst...) ?
Reading through the (SDK) code this even covers depending resources, nice!
I certainly will leave some feedback and comments in your change (https://review.opendev.org/c/openstack/python-openstackclient/+/734485). But what are your immediate plans moving forward on with this then, Artem?
There is a little todo list in the description on your change .. is there anything you yourself know that is still missing before taking this to a full review and finally merging it?
Only code that is shipped and then actively used will improve further and people will notice other required functionality or switches for later iterations. With the current state of having a somewhat working but unmaintained ospurge and a non feature complete "project purge" (supports only Block Storage v1, v2; Compute v2; Image v1, v2) this will only cause people to start hacking away on the ospurge codebase or worse building their own tools and scripts to implement project cleanup for their environments over and over again.
Regards,
Christian
Hi Christian.
Actually the patch stuck due to lack of reviewers. Idea here was not to replace “openstack project purge”, but to add a totally new implementation (hopefully later dropping project purge as such). From my POV at the moment there is nothing else I was thinking to mandatorily implement on OSC side (sure, for future I would like to give possibility to limit services to cleanup, to add cleanup of key-airs, etc). SDK part is completely independent of that. Here we definitely need to add dropping of private images. Also on DNS side we can do cleanup. Other services are tricky (while swift we can still implement relatively easy).
All in all - we can merge the PR in it’s current form (assuming we get some positive reviews).
BG, Artem
On 19. Jan 2021, at 13:33, Christian Rohmann christian.rohmann@inovex.de wrote:
Hey Artem,
thank you very much for your quick reply and pointer to the patchset you work on!
On 18/01/2021 20:14, Artem Goncharov wrote:
Ha, thats exactly the case, the whole logic sits in sdk and is spread across the supported services:
- https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/compu... https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/compute/v2/_proxy.py#L1798 - for compute. KeyPairs not dropped, since they belong to user, and not to the “project”;
- https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/block... https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/block_storage/v3/_proxy.py#L547 - block storage;
- https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/orche... https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/orchestration/v1/_proxy.py#L490
- https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/netwo... https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/network/v2/_proxy.py#L4130 - the most complex one in order to give possibility to clean “old” resource without destroying everything else
Adding image is few lines of code (never had enough time to add it), identity is a bit tricky, since also here mostly resources does not belong to Project. DNS would be also easy to do. OSC here is only providing I/F, while the logic sits in SDK and can be very easy extended for other services.
On 18. Jan 2021, at 19:52, Thomas Goirand <zigo@debian.org mailto:zigo@debian.org> wrote:
On 1/18/21 6:56 PM, Artem Goncharov wrote:
What do you mean it doesn’t implement anything at all? It does clean up compute, network, block_storage, orchestrate resources. Moreover it gives you possibility to clean “old” resources (created before or last updated before).
Oh really? With that few lines of code? I'll re-read the patch then, sorry for my bad assumptions.
Can you point at the part that's actually deleting the resources?
If I understood correctly, the cleanup relies on the SDK functionality / requirement for each resource type to provide a corresponding function( https://github.com/openstack/openstacksdk/blob/master/openstack/cloud/openst... https://github.com/openstack/openstacksdk/blob/master/openstack/cloud/openstackcloud.py#L762) ?
Reading through the (SDK) code this even covers depending resources, nice!
I certainly will leave some feedback and comments in your change (https://review.opendev.org/c/openstack/python-openstackclient/+/734485 https://review.opendev.org/c/openstack/python-openstackclient/+/734485). But what are your immediate plans moving forward on with this then, Artem?
There is a little todo list in the description on your change .. is there anything you yourself know that is still missing before taking this to a full review and finally merging it?
Only code that is shipped and then actively used will improve further and people will notice other required functionality or switches for later iterations. With the current state of having a somewhat working but unmaintained ospurge and a non feature complete "project purge" (supports only Block Storage v1, v2; Compute v2; Image v1, v2) this will only cause people to start hacking away on the ospurge codebase or worse building their own tools and scripts to implement project cleanup for their environments over and over again.
Regards,
Christian
OH! As someone who tried to champion project termination at the API layer as a community goal during the Berlin summit, I love that someone ran with our idea of implementing it in the SDK as a starting point!
I still think we should champion a purge API for each service, but the SDK is definitely the place to handle dependencies, and I love that you are doing it in threads!
I'd be happy to review/test any code related to this both in the OSC and the SDK. I sadly haven't had time to help implement anything like this, but I will somehow find the time to review/test! And I look forward to one day throwing away our internal termination logic in favor of what's in the SDK!
On 20/01/21 2:16 am, Artem Goncharov wrote:
Hi Christian.
Actually the patch stuck due to lack of reviewers. Idea here was not to replace “openstack project purge”, but to add a totally new implementation (hopefully later dropping project purge as such). From my POV at the moment there is nothing else I was thinking to mandatorily implement on OSC side (sure, for future I would like to give possibility to limit services to cleanup, to add cleanup of key-airs, etc). SDK part is completely independent of that. Here we definitely need to add dropping of private images. Also on DNS side we can do cleanup. Other services are tricky (while swift we can still implement relatively easy).
All in all - we can merge the PR in it’s current form (assuming we get some positive reviews).
BG, Artem
On 19. Jan 2021, at 13:33, Christian Rohmann <christian.rohmann@inovex.de mailto:christian.rohmann@inovex.de> wrote:
Hey Artem,
thank you very much for your quick reply and pointer to the patchset you work on!
On 18/01/2021 20:14, Artem Goncharov wrote:
Ha, thats exactly the case, the whole logic sits in sdk and is spread across the supported services:
https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/compu... https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/compute/v2/_proxy.py#L1798 - for compute. KeyPairs not dropped, since they belong to user, and not to the “project”;
https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/block... https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/block_storage/v3/_proxy.py#L547 - block storage;
https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/orche... https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/orchestration/v1/_proxy.py#L490
https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/netwo... https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/network/v2/_proxy.py#L4130 - the most complex one in order to give possibility to clean “old” resource without destroying everything else
Adding image is few lines of code (never had enough time to add it), identity is a bit tricky, since also here mostly resources does not belong to Project. DNS would be also easy to do. OSC here is only providing I/F, while the logic sits in SDK and can be very easy extended for other services.
On 18. Jan 2021, at 19:52, Thomas Goirand <zigo@debian.org mailto:zigo@debian.org> wrote:
On 1/18/21 6:56 PM, Artem Goncharov wrote:
What do you mean it doesn’t implement anything at all? It does clean up compute, network, block_storage, orchestrate resources. Moreover it gives you possibility to clean “old” resources (created before or last updated before).
Oh really? With that few lines of code? I'll re-read the patch then, sorry for my bad assumptions.
Can you point at the part that's actually deleting the resources?
If I understood correctly, the cleanup relies on the SDK functionality / requirement for each resource type to provide a corresponding function( https://github.com/openstack/openstacksdk/blob/master/openstack/cloud/openst...) ?
Reading through the (SDK) code this even covers depending resources, nice!
I certainly will leave some feedback and comments in your change (https://review.opendev.org/c/openstack/python-openstackclient/+/734485). But what are your immediate plans moving forward on with this then, Artem?
There is a little todo list in the description on your change .. is there anything you yourself know that is still missing before taking this to a full review and finally merging it?
Only code that is shipped and then actively used will improve further and people will notice other required functionality or switches for later iterations. With the current state of having a somewhat working but unmaintained ospurge and a non feature complete "project purge" (supports only Block Storage v1, v2; Compute v2; Image v1, v2) this will only cause people to start hacking away on the ospurge codebase or worse building their own tools and scripts to implement project cleanup for their environments over and over again.
Regards,
Christian
That reminded me of my old osclean https://github.com/ssbarnea/harem/blob/master/bin/osclean bash script which did some work in parallel, but I would see the purge as a very useful sdk feature. -- /sorin
On 2 Feb 2021 at 07:40:22, Adrian Turjak adriant@catalystcloud.nz wrote:
OH! As someone who tried to champion project termination at the API layer as a community goal during the Berlin summit, I love that someone ran with our idea of implementing it in the SDK as a starting point!
I still think we should champion a purge API for each service, but the SDK is definitely the place to handle dependencies, and I love that you are doing it in threads!
I'd be happy to review/test any code related to this both in the OSC and the SDK. I sadly haven't had time to help implement anything like this, but I will somehow find the time to review/test! And I look forward to one day throwing away our internal termination logic in favor of what's in the SDK!
On 20/01/21 2:16 am, Artem Goncharov wrote:
Hi Christian.
Actually the patch stuck due to lack of reviewers. Idea here was not to replace “openstack project purge”, but to add a totally new implementation (hopefully later dropping project purge as such). From my POV at the moment there is nothing else I was thinking to mandatorily implement on OSC side (sure, for future I would like to give possibility to limit services to cleanup, to add cleanup of key-airs, etc). SDK part is completely independent of that. Here we definitely need to add dropping of private images. Also on DNS side we can do cleanup. Other services are tricky (while swift we can still implement relatively easy).
All in all - we can merge the PR in it’s current form (assuming we get some positive reviews).
BG, Artem
On 19. Jan 2021, at 13:33, Christian Rohmann christian.rohmann@inovex.de wrote:
Hey Artem,
thank you very much for your quick reply and pointer to the patchset you work on!
On 18/01/2021 20:14, Artem Goncharov wrote:
Ha, thats exactly the case, the whole logic sits in sdk and is spread across the supported services:
https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/compu... - for compute. KeyPairs not dropped, since they belong to user, and not to the “project”;
https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/block... - block storage;
https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/orche...
https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/netwo... - the most complex one in order to give possibility to clean “old” resource without destroying everything else
Adding image is few lines of code (never had enough time to add it), identity is a bit tricky, since also here mostly resources does not belong to Project. DNS would be also easy to do. OSC here is only providing I/F, while the logic sits in SDK and can be very easy extended for other services.
On 18. Jan 2021, at 19:52, Thomas Goirand zigo@debian.org wrote:
On 1/18/21 6:56 PM, Artem Goncharov wrote:
What do you mean it doesn’t implement anything at all? It does clean up compute, network, block_storage, orchestrate resources. Moreover it gives you possibility to clean “old” resources (created before or last updated before).
Oh really? With that few lines of code? I'll re-read the patch then, sorry for my bad assumptions.
Can you point at the part that's actually deleting the resources?
If I understood correctly, the cleanup relies on the SDK functionality / requirement for each resource type to provide a corresponding function( https://github.com/openstack/openstacksdk/blob/master/openstack/cloud/openst...) ?
Reading through the (SDK) code this even covers depending resources, nice!
I certainly will leave some feedback and comments in your change ( https://review.opendev.org/c/openstack/python-openstackclient/+/734485). But what are your immediate plans moving forward on with this then, Artem?
There is a little todo list in the description on your change .. is there anything you yourself know that is still missing before taking this to a full review and finally merging it?
Only code that is shipped and then actively used will improve further and people will notice other required functionality or switches for later iterations. With the current state of having a somewhat working but unmaintained ospurge and a non feature complete "project purge" (supports only Block Storage v1, v2; Compute v2; Image v1, v2) this will only cause people to start hacking away on the ospurge codebase or worse building their own tools and scripts to implement project cleanup for their environments over and over again.
Regards,
Christian
Hello again,
I just ran into the openstack-resource-manager (https://github.com/SovereignCloudStack/openstack-resource-manager) apparently part of the SovereignCloudStack - SCS effort out of the Gaia-X initiative.
That tool seems to solve once again the need to clean up prior to a projects deletion.
Regards
Christian
Hi,
This is just a tiny subset of OpenStack resources with no possible flexibility vs native implementation in SDK/CLI
Artem
On 9. Mar 2021, at 14:17, Christian Rohmann christian.rohmann@inovex.de wrote:
Hello again,
I just ran into the openstack-resource-manager (https://github.com/SovereignCloudStack/openstack-resource-manager) apparently part of the SovereignCloudStack - SCS effort out of the Gaia-X initiative.
That tool seems to solve once again the need to clean up prior to a projects deletion.
Regards
Christian
Hey Artem,
On 09/03/2021 14:23, Artem Goncharov wrote:
This is just a tiny subset of OpenStack resources with no possible flexibility vs native implementation in SDK/CLI
I totally agree. I was just saying that by not having a cleanup approach provided by OpenStack there will just be more and more tools popping up, each solving the same issue ... and never being tested and updated along with OpenStack releases and new project / resources being added.
So thanks again for your work on https://review.opendev.org/c/openstack/python-openstackclient/+/734485 !
Regards
Christian
Hello Artem, all,
On 10/03/2021 15:21, Christian Rohmann wrote:
Hey Artem,
On 09/03/2021 14:23, Artem Goncharov wrote:
This is just a tiny subset of OpenStack resources with no possible flexibility vs native implementation in SDK/CLI
I totally agree. I was just saying that by not having a cleanup approach provided by OpenStack there will just be more and more tools popping up, each solving the same issue ... and never being tested and updated along with OpenStack releases and new project / resources being added.
So thanks again for your work on https://review.opendev.org/c/openstack/python-openstackclient/+/734485 !
With your change now merged I believe there should be a list of missing features / projects not yet integrated. Maybe even a blueprint (for others) to pick up an and to continue the path to provide a full solution for project cleanup. Something apparently not on this list yet is object storage, so Swift / RADOS GW (S3), ..
Swift documents something called the account reaper (https://docs.openstack.org/swift/victoria/overview_reaper.html), but I don't know how that fits into the picture. As for S3 powered by i.e. RADOS GW I don't know if there are any standard-tools yet.
Regards and thanks again for cleaning up the trash ;-)
Christian
participants (5)
-
Adrian Turjak
-
Artem Goncharov
-
Christian Rohmann
-
Sorin Sbarnea
-
Thomas Goirand