<html><head><style>body{font-family:Helvetica,Arial;font-size:13px}</style></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">On February 2, 2015 at 1:31:14 PM, Joe Gordon (<a href="mailto:joe.gordon0@gmail.com">joe.gordon0@gmail.com</a>) wrote:</div> <blockquote type="cite" class="clean_bq"><span><div><div></div><div>
<title></title>
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Feb 2, 2015 at 10:28 AM, Morgan
Fainberg <span dir="ltr"><<a href="mailto:morgan.fainberg@gmail.com" target="_blank">morgan.fainberg@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I
think the simple answer is "yes". We (keystone) should emit
notifications. And yes other projects should listen.<br>
<br>
The only thing really in discussion should be:<br>
<br>
1: soft delete or hard delete? Does the service mark it as
orphaned, or just delete (leave this to nova, cinder, etc to
discuss)<br>
<br>
2: how to cleanup when an event is missed (e.g rabbit bus goes out
to lunch).<br></blockquote>
<div><br></div>
<div><br></div>
<div>I disagree slightly, I don't think projects should directly
listen to the Keystone notifications I would rather have the API be
something from a keystone owned library, say keystonemiddleware. So
something like this:</div>
<div><br></div>
<div>from keystonemiddleware import janitor</div>
<div><br></div>
<div>keystone_janitor = janitor.Janitor()</div>
<div>keystone_janitor.register_callback(nova.tenant_cleanup)</div>
<div><br></div>
<div>keystone_janitor.spawn_greenthread()</div>
<div><br></div>
<div>That way each project doesn't have to include a lot of
boilerplate code, and keystone can easily modify/improve/upgrade
the notification mechanism.</div>
<div><br></div></div></div></div></div></div></span></blockquote><div><br></div><div>Sure. I’d place this into an implementation detail of where that actually lives. I’d be fine with that being a part of Keystone Middleware Package (probably something separate from auth_token).</div><div><br></div><div>—Morgan</div><br><blockquote type="cite" class="clean_bq"><span><div><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div> <br></div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
--Morgan<br>
<br>
Sent via mobile<br>
<div>
<div class="h5"><br>
> On Feb 2, 2015, at 10:16, Matthew Treinish <<a href="mailto:mtreinish@kortar.org">mtreinish@kortar.org</a>>
wrote:<br>
><br>
>> On Mon, Feb 02, 2015 at 11:46:53AM -0600, Matt Riedemann
wrote:<br>
>> This came up in the operators mailing list back in June
[1] but given the<br>
>> subject probably didn't get much attention.<br>
>><br>
>> Basically there is a really old bug [2] from Grizzly that
is still a problem<br>
>> and affects multiple projects. A tenant can be
deleted in Keystone even<br>
>> though other resources in other projects are under that
project, and those<br>
>> resources aren't cleaned up.<br>
><br>
> I agree this probably can be a major pain point for users.
We've had to work around it<br>
> in tempest by creating things like:<br>
><br>
> <a href="http://git.openstack.org/cgit/openstack/tempest/tree/tempest/cmd/cleanup_service.py" target="_blank">http://git.openstack.org/cgit/openstack/tempest/tree/tempest/cmd/cleanup_service.py</a><br>
> and<br>
> <a href="http://git.openstack.org/cgit/openstack/tempest/tree/tempest/cmd/cleanup.py" target="_blank">http://git.openstack.org/cgit/openstack/tempest/tree/tempest/cmd/cleanup.py</a><br>
><br>
> to ensure we aren't dangling resources after a run. But, this
doesn't work in<br>
> all cases either. (like with tenant isolation enabled)<br>
><br>
> I also know there is a stackforge project that is attempting
something similar<br>
> here:<br>
><br>
> <a href="http://git.openstack.org/cgit/stackforge/ospurge/" target="_blank">http://git.openstack.org/cgit/stackforge/ospurge/</a><br>
><br>
> It would be much nicer if the burden for doing this was taken
off users and this<br>
> was just handled cleanly under the covers.<br>
><br>
>><br>
>> Keystone implemented event notifications back in Havana
[3] but the other<br>
>> projects aren't listening on them to know when a project
has been deleted<br>
>> and act accordingly.<br>
>><br>
>> The bug has several people saying "we should talk about
this at the summit"<br>
>> for several summits, but I can't find any discussion or
summit sessions<br>
>> related back to the bug.<br>
>><br>
>> Given this is an operations and cross-project issue, I'd
like to bring it up<br>
>> again for the Vancouver summit if there is still interest
(which I'm<br>
>> assuming there is from operators).<br>
><br>
> I'd definitely support having a cross-project session on
this.<br>
><br>
>><br>
>> There is a blueprint specifically for the tenant deletion
case but it's<br>
>> targeted at only Horizon [4].<br>
>><br>
>> Is anyone still working on this? Is there sufficient
interest in a<br>
>> cross-project session at the L summit?<br>
>><br>
>> Thinking out loud, even if nova doesn't listen to events
from keystone, we<br>
>> could at least have a periodic task that looks for
instances where the<br>
>> tenant no longer exists in keystone and then take some
action (log a<br>
>> warning, shutdown/archive/, reap, etc).<br>
>><br>
>> There is also a spec for L to transfer instance ownership
[5] which could<br>
>> maybe come into play, but I wouldn't depend on it.<br>
>><br>
>> [1] <a href="http://lists.openstack.org/pipermail/openstack-operators/2014-June/004559.html" target="_blank">http://lists.openstack.org/pipermail/openstack-operators/2014-June/004559.html</a><br>
>> [2] <a href="https://bugs.launchpad.net/nova/+bug/967832" target="_blank">https://bugs.launchpad.net/nova/+bug/967832</a><br>
>> [3] <a href="https://blueprints.launchpad.net/keystone/+spec/notifications" target="_blank">https://blueprints.launchpad.net/keystone/+spec/notifications</a><br>
>> [4] <a href="https://blueprints.launchpad.net/horizon/+spec/tenant-deletion" target="_blank">https://blueprints.launchpad.net/horizon/+spec/tenant-deletion</a><br>
>> [5] <a href="https://review.openstack.org/#/c/105367/" target="_blank">https://review.openstack.org/#/c/105367/</a><br>
><br>
> -Matt Treinish<br></div>
</div>
> _______________________________________________<br>
> OpenStack-operators mailing list<br>
> <a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br>
_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
</blockquote>
</div>
<br></div>
</div>
</div></div></span></blockquote></body></html>