Open Stack

On Fri, 2014-03-14 at 08:37 +0100, Radomir Dopieralski wrote:
> Hello,
> 
> I also think that this thread is going in the wrong direction, but I
> don't think the direction Boris wants is the correct one either. Frankly
> I'm a little surprised that nobody mentioned another advantage that soft
> delete gives us, the one that I think it was actually used for originally.
> 
> You see, soft delete is an optimization. It's there to make the system
> work faster as a whole, have less code and be simpler to maintain and debug.
> 
> How does it do it, when, as clearly shown in the first post in this
> thread, it makes the queries slower, requires additional indices in the
> database and more logic in the queries?

I feel it isn't an optimization if:

* It slows down the code base
* Makes the code harder to read and understand
* Deliberately obscures the actions of removing and restoring resources
* Encourages the idea that everything in the system is "undoable", like
the cloud is a Word doc.

>  The answer is, by doing more
> with those queries, by making you write less code, execute fewer queries
> to the databases and avoid duplicating the same data in multiple places.

Fewer queries does not aklways make faster code, nor does it lead to
inherently race-free code.

> OpenStack is a big, distributed system of multiple databases that
> sometimes rely on each other and cross-reference their records. It's not
> uncommon to have some long-running operation started, that uses some
> data, and then, in the middle of its execution, have that data deleted.
> With soft delete, that's not a problem -- the operation can continue
> safely and proceed as scheduled, with the data it was started with in
> the first place -- it still has access to the deleted records as if
> nothing happened.

I believe a better solution would be to use Boris' solution and
implement safeguards around the delete operation. For instance, not
being able to delete an instance that has tasks still running against
it. Either that, or implement true task abortion logic that can notify
distributed components about the need to stop a running task because
either the user wants to delete a resource or simply cancel the
operation they began.

>  You simply won't be able to schedule another operation
> like that with the same data, because it has been soft-deleted and won't
> pass the validation at the beginning (or even won't appear in the UI or
> CLI). This solves a lot of race conditions, error handling, additional
> checks to make sure the record still exists, etc.

Sorry, I disagree here. Components that rely on the soft-delete behavior
to get the resource data from the database should instead respond to a
NotFound that gets raised by aborting their running task.

> Without soft delete, you need to write custom code every time to handle
> the case of a record being deleted mid-operation, including all the
> possible combinations of which record and when.

Not custom code. Explicit code paths for explicit actions.

>  Or you need to copy all
> the relevant data in advance over to whatever is executing that
> operation. 

This is already happening.

> This cannot be abstracted away entirely (although tools like
> TaskFlow help), as this is specific to the case you are handling. And
> it's not easy to find all the places where you can have a race condition
> like that -- especially when you are modifying existing code that has
> been relying on soft delete before. You can have bugs undetected for
> years, that only appear in production, on very large deployments, and
> are impossible to reproduce reliably.
> 
> There are more similar cases like that, including cascading deletes and
> more advanced stuff, but I think this single case already shows that
> the advantages of soft delete out-weight its disadvantages.

I respectfully disagree :) I think the benefits of explicit code paths
and increased performance of the database outweigh the costs of changing
existing code.

Best,
-jay

> On 13/03/14 19:52, Boris Pavlovic wrote:
> > Hi all, 
> > 
> > 
> > I would like to fix direction of this thread. Cause it is going in wrong
> > direction. 
> > 
> > To assume:
> > 1) Yes restoring already deleted recourses could be useful. 
> > 2) Current approach with soft deletion is broken by design and we should
> > get rid of them. 
> > 
> > More about why I think that it is broken: 
> > 1) When you are restoring some resource you should restore N records
> > from N tables (e.g. VM)
> > 2) Restoring sometimes means not only restoring DB records. 
> > 3) Not all resources should be restorable (e.g. why I need to restore
> > fixed_ip? or key-pairs?)
> > 
> > 
> > So what we should think about is:
> > 1) How to implement restoring functionally in common way (e.g. framework
> > that will be in oslo) 
> > 2) Split of work of getting rid of soft deletion in steps (that I
> > already mention):
> > a) remove soft deletion from places where we are not using it
> > b) replace internal code where we are using soft deletion to that framework 
> > c) replace API stuff using ceilometer (for logs) or this framework (for
> > restorable stuff)
> > 
> > 
> > To put in a nutshell: Restoring Delete resources / Delayed Deletion !=
> > Soft deletion. 
> > 
> > 
> > Best regards,
> > Boris Pavlovic 
> > 
> > 
> > 
> > On Thu, Mar 13, 2014 at 9:21 PM, Mike Wilson <geekinutah at gmail.com
> > <mailto:geekinutah at gmail.com>> wrote:
> > 
> >     For some guests we use the LVM imagebackend and there are times when
> >     the guest is deleted on accident. Humans, being what they are, don't
> >     back up their files and don't take care of important data, so it is
> >     not uncommon to use lvrestore and "undelete" an instance so that
> >     people can get their data. Of course, this is not always possible if
> >     the data has been subsequently overwritten. But it is common enough
> >     that I imagine most of our operators are familiar with how to do it.
> >     So I guess my saying that we do it on a regular basis is not quite
> >     accurate. Probably would be better to say that it is not uncommon to
> >     do this, but definitely not a daily task or something of that ilk.
> > 
> >     I have personally "undeleted" an instance a few times after
> >     accidental deletion also. I can't remember the specifics, but I do
> >     remember doing it :-).
> > 
> >     -Mike
> > 
> > 
> >     On Tue, Mar 11, 2014 at 12:46 PM, Johannes Erdfelt
> >     <johannes at erdfelt.com <mailto:johannes at erdfelt.com>> wrote:
> > 
> >         On Tue, Mar 11, 2014, Mike Wilson <geekinutah at gmail.com
> >         <mailto:geekinutah at gmail.com>> wrote:
> >         > Undeleting things is an important use case in my opinion. We
> >         do this in our
> >         > environment on a regular basis. In that light I'm not sure
> >         that it would be
> >         > appropriate just to log the deletion and git rid of the row. I
> >         would like
> >         > to see it go to an archival table where it is easily restored.
> > 
> >         I'm curious, what are you undeleting and why?
> > 
> >         JE
> > 
> > 
> >         _______________________________________________
> >         OpenStack-dev mailing list
> >         OpenStack-dev at lists.openstack.org
> >         <mailto:OpenStack-dev at lists.openstack.org>
> >         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> > 
> > 
> >     _______________________________________________
> >     OpenStack-dev mailing list
> >     OpenStack-dev at lists.openstack.org
> >     <mailto:OpenStack-dev at lists.openstack.org>
> >     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> > 
> > 
> > 
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev