<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Thanks for the thorough feedback Adrian.<br>
<br>
My opinion is also that Keystone should not be the actor in
executing this functionality but somewhere else<br>
whether that is Adjutant or any other form (application, library,
CLI etc).<br>
<br>
I would also like to bring up the point about knowing if a project
is "dirty" (it has provisioned resources).<br>
This is something that I think all business logic would benefit
from, we've had issue with knowing when<br>
resources should be deleted, our solution is pretty much look at
metrics the last X minutes, check if project<br>
is disabled and compare to business logic that says it should be
deleted.<br>
<br>
While the above works it kills some of logical points of disabling a
project since the only thing that knows if<br>
the project should be deleted or is actually disabled is the
business logic application that says they clicked the<br>
deleted button and not disabled.<br>
<br>
Most of the functionality you are mentioning is things that the
ospurge project has been working to implement and the<br>
maintainer even did a full rewrite which improved the dependency
arrangement for resource removal.<br>
<br>
I think the biggest win for this community goal would be the
developers of the projects would be available for input regarding<br>
the project specific code that does purging. There has been some
really nasty bugs in ospurge in the past that if executed with the
admin<br>
user you would wipe everything and not only that project, which is
probably a issue that makes people think twice about<br>
using a purging toolkit at all.<br>
<br>
We should carefully consider what parts of ospurge could be reused,
concept, code or anything in between that could help derive<br>
what direction we wan't to push this goal.<br>
<br>
I'm excited :)<br>
<br>
Best regards<br>
Tobias<br>
<br>
<div class="moz-cite-prefix">On 01/22/2019 02:18 AM, Adrian Turjak
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:a55896d9-a689-c393-cc70-f799242fd06e@catalyst.net.nz">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
I've expanded on the notes in the etherpad about why Keystone
isn't the actor.<br>
<br>
At the summit we discussed this option, and all the people
familiar with Keystone who were in the room (or in some later
discussions), agreed that making Keystone the actor is a BAD idea.<br>
<br>
Keystone does not currently do any orchestration or workflow of
this nature, making it do that adds a lot of extra logic which it
just shouldn't need. After a project delete it would need to call
all the APIs, and then confirm they succeeded, and maybe retry.
This would have to be done asynchronously since waiting and
confirming the deletion would take longer than a single API call
to delete a project in Keystone should take. That kind of logic
doesn't fit in Keystone. Not to mention there are issues on how
Keystone would know which services support such an API, and where
exactly it might be (although catalog + consistent API placement
or discovery could solve that).<br>
<br>
Essentially, going down the route of "make this Keystone's
problem" is in my opinion a hard NO, but I'll let the Keystone
devs weigh in on that before we make that a very firm hard NO.<br>
<br>
As for solutions. Ideally we do implement the APIs per service
(that's the end goal), but we ALSO make libraries that do deletion
of resource using the existing APIs. If the library sees that a
service version is one with the purge API it uses it, otherwise it
has a fallback for less efficient deletion. This has the major
benefit of working for all existing deployments, and ones stuck on
older OpenStack versions. This is a universal problem and we need
to solve it backwards AND forwards.<br>
<br>
By doing both (with a first step focus on the libraries) we can
actually give projects more time to build the purge API, and maybe
have the API portion of the goal extend into another cycle if
needed.<br>
<br>
Essentially, we'd make a purge library that uses the SDK to delete
resources. If a service has a purge endpoint, then the library
(via the SDK) uses that. The specifics of how the library purges,
or if the library will be split into multiple libraries (one top
level, and then one per service) is to be decided. <br>
<br>
A rough look at what a deletion process might looks like:<br>
1. Disable project in Keystone (so no new resources can be created
or modified), or clear all role assignments (and api-keys) from
project.<br>
2. Purge platform orchestration services (Magnum, Sahara<br>
3. Purge Heat (Heat after Magnum, because magnum and such use
Heat, and deleting Heat stacks without deleting the 'resource'
which uses that stack can leave a mess)<br>
4. Purge everything left (order to be decided or potentially
dynamically chosen).<br>
5. Delete or Disable Keystone project (disable is enough really).<br>
<br>
The actor is then first a CLI built into the purge library as a
OSClient command, then secondly maybe an API or two in Adjutant
which will use this library. Or anyone can use the library and
make anything they want an actor.<br>
<br>
Ideally if we can even make the library allow selectively choosing
which services to purge (conditional on dependency chain), that
could be useful for cases where a user wants to delete everything
except maybe what's in Swift or Cinder.<br>
<br>
<br>
This is in many ways a HUGE goal, but one that we really need to
accomplish. We've lived with this problem too long and the longer
we leave it unsolved, the harder it becomes.
<br>
<br>
<br>
<div class="moz-cite-prefix">On 22/01/19 9:30 AM, Lance Bragstad
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAE6oFcHaOGc1C1v3a3mKe4-vusfK6zPOy8N8GMVzF6DPVXViag@mail.gmail.com">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Mon, Jan 21, 2019 at 2:18 PM Ed Leafe
<<a href="mailto:ed@leafe.com" moz-do-not-send="true">ed@leafe.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
On Jan 21, 2019, at 1:55 PM, Lance Bragstad <<a
href="mailto:lbragstad@gmail.com" target="_blank"
moz-do-not-send="true">lbragstad@gmail.com</a>>
wrote:<br>
> <br>
> Are you referring to the system scope approach
detailed on line 38, here [0]?<br>
<br>
Yes.<br>
<br>
> I might be misunderstanding something, but I didn't
think keystone was going to iterate all available
services and call clean-up APIs. I think it was just
that services would be able to expose an endpoint that
cleans up resources without a project scoped token
(e.g., it would be system scoped [1]).<br>
> <br>
> [0] <a
href="https://etherpad.openstack.org/p/community-goal-project-deletion"
rel="noreferrer" target="_blank"
moz-do-not-send="true">
https://etherpad.openstack.org/p/community-goal-project-deletion</a><br>
> [1] <a
href="https://docs.openstack.org/keystone/latest/admin/tokens-overview.html#system-scoped-tokens"
rel="noreferrer" target="_blank"
moz-do-not-send="true">
https://docs.openstack.org/keystone/latest/admin/tokens-overview.html#system-scoped-tokens</a>
<br>
<br>
It is more likely that I’m misunderstanding. Reading
that etherpad, it appeared that it was indeed the goal
to have project deletion in Keystone cascade to all the
services, but I guess I missed line 19.<br>
<br>
So if it isn’t Keystone calling this API on all the
services, what would be the appropriate actor?<br>
</blockquote>
<div><br>
</div>
<div>The actor could still be something like os-purge or
adjutant [0]. Depending on how the implementation shakes
out in each service, the implementation in the actor
could be an interation of all services calling the same
API for each one. I guess the benefit is that the actor
doesn't need to manage the deletion order based on the
dependencies of the resources (internal or external to a
service).</div>
<div><br>
</div>
<div>Adrian, and others, have given this a bunch more
thought than I have. So I'm curious to hear if what I'm
saying is in line with how they've envisioned things.
I'm recalling most of this from Berlin.</div>
<div><br>
</div>
<div>[0] <a
href="https://adjutant.readthedocs.io/en/latest/"
moz-do-not-send="true">https://adjutant.readthedocs.io/en/latest/</a></div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<br>
<br>
-- Ed Leafe<br>
<br>
<br>
<br>
<br>
<br>
</blockquote>
</div>
</div>
</div>
</blockquote>
</blockquote>
<br>
</body>
</html>