Open Stack

A few inline comments and a general point

How do we handle scenarios like volumes when we have a per-component janitor rather than a single co-ordinator ?

To be clean,

1. nova should shutdown the instance
2. nova should then ask the volume to be detached
3. cinder could then perform the 'project deletion' action as configured by the operator (such as shelve or backup)
4. nova could then perform the 'project deletion' action as configured by the operator (such as VM delete or shelve)

If we have both cinder and nova responding to a single message, cinder would do 3. Immediately and nova would be doing the shutdown which is likely to lead to a volume which could not be shelved cleanly.

The problem I see with messages is that co-ordination of the actions may require ordering between the components.  The disable/enable cases would show this in a worse scenario.

Tim

> -----Original Message-----
> From: Ian Cordasco [mailto:ian.cordasco at RACKSPACE.COM]
> Sent: 19 February 2015 17:49
> To: OpenStack Development Mailing List (not for usage questions); Joe Gordon
> Cc: openstack-operators at lists.openstack.org
> Subject: Re: [Openstack-operators] [openstack-dev] Resources owned by a
> project/tenant are not cleaned up after that project is deleted from keystone
> 
> 
> 
> On 2/2/15, 15:41, "Morgan Fainberg" <morgan.fainberg at gmail.com> wrote:
> 
> >
> >On February 2, 2015 at 1:31:14 PM, Joe Gordon (joe.gordon0 at gmail.com)
> >wrote:
> >
> >
> >
> >On Mon, Feb 2, 2015 at 10:28 AM, Morgan Fainberg
> ><morgan.fainberg at gmail.com> wrote:
> >
> >I think the simple answer is "yes". We (keystone) should emit
> >notifications. And yes other projects should listen.
> >
> >The only thing really in discussion should be:
> >
> >1: soft delete or hard delete? Does the service mark it as orphaned, or
> >just delete (leave this to nova, cinder, etc to discuss)
> >
> >2: how to cleanup when an event is missed (e.g rabbit bus goes out to
> >lunch).
> >
> >
> >
> >
> >
> >
> >I disagree slightly, I don't think projects should directly listen to
> >the Keystone notifications I would rather have the API be something
> >from a keystone owned library, say keystonemiddleware. So something like
> this:
> >
> >
> >from keystonemiddleware import janitor
> >
> >
> >keystone_janitor = janitor.Janitor()
> >keystone_janitor.register_callback(nova.tenant_cleanup)
> >
> >
> >keystone_janitor.spawn_greenthread()
> >
> >
> >That way each project doesn't have to include a lot of boilerplate
> >code, and keystone can easily modify/improve/upgrade the notification
> mechanism.
> >
> >


I assume janitor functions can be used for

- enable/disable project
- enable/disable user

> >
> >
> >
> >
> >
> >
> >
> >
> >
> >Sure. I’d place this into an implementation detail of where that
> >actually lives. I’d be fine with that being a part of Keystone
> >Middleware Package (probably something separate from auth_token).
> >
> >
> >—Morgan
> >
> 
> I think my only concern is what should other projects do and how much do we
> want to allow operators to configure this? I can imagine it being preferable to
> have safe (without losing much data) policies for this as a default and to allow
> operators to configure more destructive policies as part of deploying certain
> services.
> 

Depending on the cloud, an operator could want different semantics for delete project's impact, between delete or 'shelve' style or maybe disable.

> 
> >
> >
> >
> >
> >
> >--Morgan
> >
> >Sent via mobile
> >
> >> On Feb 2, 2015, at 10:16, Matthew Treinish <mtreinish at kortar.org> wrote:
> >>
> >>> On Mon, Feb 02, 2015 at 11:46:53AM -0600, Matt Riedemann wrote:
> >>> This came up in the operators mailing list back in June [1] but
> >>>given the  subject probably didn't get much attention.
> >>>
> >>> Basically there is a really old bug [2] from Grizzly that is still a
> >>>problem  and affects multiple projects.  A tenant can be deleted in
> >>>Keystone even  though other resources in other projects are under
> >>>that project, and those  resources aren't cleaned up.
> >>
> >> I agree this probably can be a major pain point for users. We've had
> >>to work around it  in tempest by creating things like:
> >>
> >>
> >http://git.openstack.org/cgit/openstack/tempest/tree/tempest/cmd/cleanu
> >p_s
> >ervice.py
> ><http://git.openstack.org/cgit/openstack/tempest/tree/tempest/cmd/clean
> >up_
> >service.py>
> >> and
> >>
> >http://git.openstack.org/cgit/openstack/tempest/tree/tempest/cmd/cleanu
> >p.p
> >y
> ><http://git.openstack.org/cgit/openstack/tempest/tree/tempest/cmd/cleanup
> .
> >py>
> >>
> >> to ensure we aren't dangling resources after a run. But, this doesn't
> >>work in  all cases either. (like with tenant isolation enabled)
> >>
> >> I also know there is a stackforge project that is attempting
> >>something similar
> >> here:
> >>
> >> http://git.openstack.org/cgit/stackforge/ospurge/
> >>
> >> It would be much nicer if the burden for doing this was taken off
> >>users and this  was just handled cleanly under the covers.
> >>
> >>>
> >>> Keystone implemented event notifications back in Havana [3] but the
> >>>other  projects aren't listening on them to know when a project has
> >>>been deleted  and act accordingly.
> >>>
> >>> The bug has several people saying "we should talk about this at the
> >>>summit"
> >>> for several summits, but I can't find any discussion or summit
> >>>sessions  related back to the bug.
> >>>
> >>> Given this is an operations and cross-project issue, I'd like to
> >>>bring it up  again for the Vancouver summit if there is still
> >>>interest (which I'm  assuming there is from operators).
> >>
> >> I'd definitely support having a cross-project session on this.
> >>
> >>>
> >>> There is a blueprint specifically for the tenant deletion case but
> >>> it's targeted at only Horizon [4].
> >>>
> >>> Is anyone still working on this? Is there sufficient interest in a
> >>> cross-project session at the L summit?
> >>>
> >>> Thinking out loud, even if nova doesn't listen to events from
> >>>keystone, we  could at least have a periodic task that looks for
> >>>instances where the  tenant no longer exists in keystone and then
> >>>take some action (log a  warning, shutdown/archive/, reap, etc).
> >>>
> >>> There is also a spec for L to transfer instance ownership [5] which
> >>>could  maybe come into play, but I wouldn't depend on it.
> >>>
> >>> [1]
> >http://lists.openstack.org/pipermail/openstack-operators/2014-June/004559.
> >html
> ><http://lists.openstack.org/pipermail/openstack-operators/2014-June/004
> >559
> >.html>
> >>> [2] https://bugs.launchpad.net/nova/+bug/967832
> >>> [3]
> >https://blueprints.launchpad.net/keystone/+spec/notifications
> ><https://blueprints.launchpad.net/keystone/+spec/notifications>
> >>> [4]
> >https://blueprints.launchpad.net/horizon/+spec/tenant-deletion
> ><https://blueprints.launchpad.net/horizon/+spec/tenant-deletion>
> >>> [5] https://review.openstack.org/#/c/105367/
> >>
> >> -Matt Treinish
> >
> >
> >> _______________________________________________
> >> OpenStack-operators mailing list
> >> OpenStack-operators at lists.openstack.org
> >>
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> ><http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operator
> >s>
> >
> >_______________________________________________
> >OpenStack-operators mailing list
> >OpenStack-operators at lists.openstack.org
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> 
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators