<div dir="ltr">+1 to what Jay says here. This hidden behavior moistly just causes problems and allows hacking hidden ways to restore things.<div><br></div><div>-Mike</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Fri, Mar 14, 2014 at 9:55 AM, Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="">On Fri, 2014-03-14 at 08:37 +0100, Radomir Dopieralski wrote:<br>
> Hello,<br>
><br>
> I also think that this thread is going in the wrong direction, but I<br>
> don't think the direction Boris wants is the correct one either. Frankly<br>
> I'm a little surprised that nobody mentioned another advantage that soft<br>
> delete gives us, the one that I think it was actually used for originally.<br>
><br>
> You see, soft delete is an optimization. It's there to make the system<br>
> work faster as a whole, have less code and be simpler to maintain and debug.<br>
><br>
> How does it do it, when, as clearly shown in the first post in this<br>
> thread, it makes the queries slower, requires additional indices in the<br>
> database and more logic in the queries?<br>
<br>
</div>I feel it isn't an optimization if:<br>
<br>
* It slows down the code base<br>
* Makes the code harder to read and understand<br>
* Deliberately obscures the actions of removing and restoring resources<br>
* Encourages the idea that everything in the system is "undoable", like<br>
the cloud is a Word doc.<br>
<div class=""><br>
> The answer is, by doing more<br>
> with those queries, by making you write less code, execute fewer queries<br>
> to the databases and avoid duplicating the same data in multiple places.<br>
<br>
</div>Fewer queries does not aklways make faster code, nor does it lead to<br>
inherently race-free code.<br>
<div class=""><br>
> OpenStack is a big, distributed system of multiple databases that<br>
> sometimes rely on each other and cross-reference their records. It's not<br>
> uncommon to have some long-running operation started, that uses some<br>
> data, and then, in the middle of its execution, have that data deleted.<br>
> With soft delete, that's not a problem -- the operation can continue<br>
> safely and proceed as scheduled, with the data it was started with in<br>
> the first place -- it still has access to the deleted records as if<br>
> nothing happened.<br>
<br>
</div>I believe a better solution would be to use Boris' solution and<br>
implement safeguards around the delete operation. For instance, not<br>
being able to delete an instance that has tasks still running against<br>
it. Either that, or implement true task abortion logic that can notify<br>
distributed components about the need to stop a running task because<br>
either the user wants to delete a resource or simply cancel the<br>
operation they began.<br>
<div class=""><br>
> You simply won't be able to schedule another operation<br>
> like that with the same data, because it has been soft-deleted and won't<br>
> pass the validation at the beginning (or even won't appear in the UI or<br>
> CLI). This solves a lot of race conditions, error handling, additional<br>
> checks to make sure the record still exists, etc.<br>
<br>
</div>Sorry, I disagree here. Components that rely on the soft-delete behavior<br>
to get the resource data from the database should instead respond to a<br>
NotFound that gets raised by aborting their running task.<br>
<div class=""><br>
> Without soft delete, you need to write custom code every time to handle<br>
> the case of a record being deleted mid-operation, including all the<br>
> possible combinations of which record and when.<br>
<br>
</div>Not custom code. Explicit code paths for explicit actions.<br>
<div class=""><br>
> Or you need to copy all<br>
> the relevant data in advance over to whatever is executing that<br>
> operation.<br>
<br>
</div>This is already happening.<br>
<div class=""><br>
> This cannot be abstracted away entirely (although tools like<br>
> TaskFlow help), as this is specific to the case you are handling. And<br>
> it's not easy to find all the places where you can have a race condition<br>
> like that -- especially when you are modifying existing code that has<br>
> been relying on soft delete before. You can have bugs undetected for<br>
> years, that only appear in production, on very large deployments, and<br>
> are impossible to reproduce reliably.<br>
><br>
> There are more similar cases like that, including cascading deletes and<br>
> more advanced stuff, but I think this single case already shows that<br>
> the advantages of soft delete out-weight its disadvantages.<br>
<br>
</div>I respectfully disagree :) I think the benefits of explicit code paths<br>
and increased performance of the database outweigh the costs of changing<br>
existing code.<br>
<br>
Best,<br>
-jay<br>
<div class="HOEnZb"><div class="h5"><br>
> On 13/03/14 19:52, Boris Pavlovic wrote:<br>
> > Hi all,<br>
> ><br>
> ><br>
> > I would like to fix direction of this thread. Cause it is going in wrong<br>
> > direction.<br>
> ><br>
> > To assume:<br>
> > 1) Yes restoring already deleted recourses could be useful.<br>
> > 2) Current approach with soft deletion is broken by design and we should<br>
> > get rid of them.<br>
> ><br>
> > More about why I think that it is broken:<br>
> > 1) When you are restoring some resource you should restore N records<br>
> > from N tables (e.g. VM)<br>
> > 2) Restoring sometimes means not only restoring DB records.<br>
> > 3) Not all resources should be restorable (e.g. why I need to restore<br>
> > fixed_ip? or key-pairs?)<br>
> ><br>
> ><br>
> > So what we should think about is:<br>
> > 1) How to implement restoring functionally in common way (e.g. framework<br>
> > that will be in oslo)<br>
> > 2) Split of work of getting rid of soft deletion in steps (that I<br>
> > already mention):<br>
> > a) remove soft deletion from places where we are not using it<br>
> > b) replace internal code where we are using soft deletion to that framework<br>
> > c) replace API stuff using ceilometer (for logs) or this framework (for<br>
> > restorable stuff)<br>
> ><br>
> ><br>
> > To put in a nutshell: Restoring Delete resources / Delayed Deletion !=<br>
> > Soft deletion.<br>
> ><br>
> ><br>
> > Best regards,<br>
> > Boris Pavlovic<br>
> ><br>
> ><br>
> ><br>
> > On Thu, Mar 13, 2014 at 9:21 PM, Mike Wilson <<a href="mailto:geekinutah@gmail.com">geekinutah@gmail.com</a><br>
> > <mailto:<a href="mailto:geekinutah@gmail.com">geekinutah@gmail.com</a>>> wrote:<br>
> ><br>
> > For some guests we use the LVM imagebackend and there are times when<br>
> > the guest is deleted on accident. Humans, being what they are, don't<br>
> > back up their files and don't take care of important data, so it is<br>
> > not uncommon to use lvrestore and "undelete" an instance so that<br>
> > people can get their data. Of course, this is not always possible if<br>
> > the data has been subsequently overwritten. But it is common enough<br>
> > that I imagine most of our operators are familiar with how to do it.<br>
> > So I guess my saying that we do it on a regular basis is not quite<br>
> > accurate. Probably would be better to say that it is not uncommon to<br>
> > do this, but definitely not a daily task or something of that ilk.<br>
> ><br>
> > I have personally "undeleted" an instance a few times after<br>
> > accidental deletion also. I can't remember the specifics, but I do<br>
> > remember doing it :-).<br>
> ><br>
> > -Mike<br>
> ><br>
> ><br>
> > On Tue, Mar 11, 2014 at 12:46 PM, Johannes Erdfelt<br>
> > <<a href="mailto:johannes@erdfelt.com">johannes@erdfelt.com</a> <mailto:<a href="mailto:johannes@erdfelt.com">johannes@erdfelt.com</a>>> wrote:<br>
> ><br>
> > On Tue, Mar 11, 2014, Mike Wilson <<a href="mailto:geekinutah@gmail.com">geekinutah@gmail.com</a><br>
> > <mailto:<a href="mailto:geekinutah@gmail.com">geekinutah@gmail.com</a>>> wrote:<br>
> > > Undeleting things is an important use case in my opinion. We<br>
> > do this in our<br>
> > > environment on a regular basis. In that light I'm not sure<br>
> > that it would be<br>
> > > appropriate just to log the deletion and git rid of the row. I<br>
> > would like<br>
> > > to see it go to an archival table where it is easily restored.<br>
> ><br>
> > I'm curious, what are you undeleting and why?<br>
> ><br>
> > JE<br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > OpenStack-dev mailing list<br>
> > <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> > <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > OpenStack-dev mailing list<br>
> > <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> > <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > OpenStack-dev mailing list<br>
> > <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>