<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>Off topic but, I'd like to see a word doc written out with the history of the cloud, that'd be pretty sweet.</div>
<div><br>
</div>
<div>Especially if its something like google docs where u can watch the changes happen in realtime.</div>
<div><br>
</div>
<div>+2</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Jay Pipes <<a href="mailto:jaypipes@gmail.com">jaypipes@gmail.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Friday, March 14, 2014 at 7:55 AM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)<br>
</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div>
<div>On Fri, 2014-03-14 at 08:37 +0100, Radomir Dopieralski wrote:</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>Hello,</div>
<div></div>
<div>I also think that this thread is going in the wrong direction, but I</div>
<div>don't think the direction Boris wants is the correct one either. Frankly</div>
<div>I'm a little surprised that nobody mentioned another advantage that soft</div>
<div>delete gives us, the one that I think it was actually used for originally.</div>
<div></div>
<div>You see, soft delete is an optimization. It's there to make the system</div>
<div>work faster as a whole, have less code and be simpler to maintain and debug.</div>
<div></div>
<div>How does it do it, when, as clearly shown in the first post in this</div>
<div>thread, it makes the queries slower, requires additional indices in the</div>
<div>database and more logic in the queries?</div>
</blockquote>
<div><br>
</div>
<div>I feel it isn't an optimization if:</div>
<div><br>
</div>
<div>* It slows down the code base</div>
<div>* Makes the code harder to read and understand</div>
<div>* Deliberately obscures the actions of removing and restoring resources</div>
<div>* Encourages the idea that everything in the system is "undoable", like</div>
<div>the cloud is a Word doc.</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div> The answer is, by doing more</div>
<div>with those queries, by making you write less code, execute fewer queries</div>
<div>to the databases and avoid duplicating the same data in multiple places.</div>
</blockquote>
<div><br>
</div>
<div>Fewer queries does not aklways make faster code, nor does it lead to</div>
<div>inherently race-free code.</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>OpenStack is a big, distributed system of multiple databases that</div>
<div>sometimes rely on each other and cross-reference their records. It's not</div>
<div>uncommon to have some long-running operation started, that uses some</div>
<div>data, and then, in the middle of its execution, have that data deleted.</div>
<div>With soft delete, that's not a problem -- the operation can continue</div>
<div>safely and proceed as scheduled, with the data it was started with in</div>
<div>the first place -- it still has access to the deleted records as if</div>
<div>nothing happened.</div>
</blockquote>
<div><br>
</div>
<div>I believe a better solution would be to use Boris' solution and</div>
<div>implement safeguards around the delete operation. For instance, not</div>
<div>being able to delete an instance that has tasks still running against</div>
<div>it. Either that, or implement true task abortion logic that can notify</div>
<div>distributed components about the need to stop a running task because</div>
<div>either the user wants to delete a resource or simply cancel the</div>
<div>operation they began.</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div> You simply won't be able to schedule another operation</div>
<div>like that with the same data, because it has been soft-deleted and won't</div>
<div>pass the validation at the beginning (or even won't appear in the UI or</div>
<div>CLI). This solves a lot of race conditions, error handling, additional</div>
<div>checks to make sure the record still exists, etc.</div>
</blockquote>
<div><br>
</div>
<div>Sorry, I disagree here. Components that rely on the soft-delete behavior</div>
<div>to get the resource data from the database should instead respond to a</div>
<div>NotFound that gets raised by aborting their running task.</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>Without soft delete, you need to write custom code every time to handle</div>
<div>the case of a record being deleted mid-operation, including all the</div>
<div>possible combinations of which record and when.</div>
</blockquote>
<div><br>
</div>
<div>Not custom code. Explicit code paths for explicit actions.</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div> Or you need to copy all</div>
<div>the relevant data in advance over to whatever is executing that</div>
<div>operation. </div>
</blockquote>
<div><br>
</div>
<div>This is already happening.</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>This cannot be abstracted away entirely (although tools like</div>
<div>TaskFlow help), as this is specific to the case you are handling. And</div>
<div>it's not easy to find all the places where you can have a race condition</div>
<div>like that -- especially when you are modifying existing code that has</div>
<div>been relying on soft delete before. You can have bugs undetected for</div>
<div>years, that only appear in production, on very large deployments, and</div>
<div>are impossible to reproduce reliably.</div>
<div></div>
<div>There are more similar cases like that, including cascading deletes and</div>
<div>more advanced stuff, but I think this single case already shows that</div>
<div>the advantages of soft delete out-weight its disadvantages.</div>
</blockquote>
<div><br>
</div>
<div>I respectfully disagree :) I think the benefits of explicit code paths</div>
<div>and increased performance of the database outweigh the costs of changing</div>
<div>existing code.</div>
<div><br>
</div>
<div>Best,</div>
<div>-jay</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>On 13/03/14 19:52, Boris Pavlovic wrote:</div>
<div>> Hi all, </div>
<div>> </div>
<div>> </div>
<div>> I would like to fix direction of this thread. Cause it is going in wrong</div>
<div>> direction. </div>
<div>> </div>
<div>> To assume:</div>
<div>> 1) Yes restoring already deleted recourses could be useful. </div>
<div>> 2) Current approach with soft deletion is broken by design and we should</div>
<div>> get rid of them. </div>
<div>> </div>
<div>> More about why I think that it is broken: </div>
<div>> 1) When you are restoring some resource you should restore N records</div>
<div>> from N tables (e.g. VM)</div>
<div>> 2) Restoring sometimes means not only restoring DB records. </div>
<div>> 3) Not all resources should be restorable (e.g. why I need to restore</div>
<div>> fixed_ip? or key-pairs?)</div>
<div>> </div>
<div>> </div>
<div>> So what we should think about is:</div>
<div>> 1) How to implement restoring functionally in common way (e.g. framework</div>
<div>> that will be in oslo) </div>
<div>> 2) Split of work of getting rid of soft deletion in steps (that I</div>
<div>> already mention):</div>
<div>> a) remove soft deletion from places where we are not using it</div>
<div>> b) replace internal code where we are using soft deletion to that framework
</div>
<div>> c) replace API stuff using ceilometer (for logs) or this framework (for</div>
<div>> restorable stuff)</div>
<div>> </div>
<div>> </div>
<div>> To put in a nutshell: Restoring Delete resources / Delayed Deletion !=</div>
<div>> Soft deletion. </div>
<div>> </div>
<div>> </div>
<div>> Best regards,</div>
<div>> Boris Pavlovic </div>
<div>> </div>
<div>> </div>
<div>> </div>
<div>> On Thu, Mar 13, 2014 at 9:21 PM, Mike Wilson <<a href="mailto:geekinutah@gmail.com">geekinutah@gmail.com</a></div>
<div>> <<a href="mailto:geekinutah@gmail.com>">mailto:geekinutah@gmail.com></a>> wrote:</div>
<div>> </div>
<div>> For some guests we use the LVM imagebackend and there are times when</div>
<div>> the guest is deleted on accident. Humans, being what they are, don't</div>
<div>> back up their files and don't take care of important data, so it is</div>
<div>> not uncommon to use lvrestore and "undelete" an instance so that</div>
<div>> people can get their data. Of course, this is not always possible if</div>
<div>> the data has been subsequently overwritten. But it is common enough</div>
<div>> that I imagine most of our operators are familiar with how to do it.</div>
<div>> So I guess my saying that we do it on a regular basis is not quite</div>
<div>> accurate. Probably would be better to say that it is not uncommon to</div>
<div>> do this, but definitely not a daily task or something of that ilk.</div>
<div>> </div>
<div>> I have personally "undeleted" an instance a few times after</div>
<div>> accidental deletion also. I can't remember the specifics, but I do</div>
<div>> remember doing it :-).</div>
<div>> </div>
<div>> -Mike</div>
<div>> </div>
<div>> </div>
<div>> On Tue, Mar 11, 2014 at 12:46 PM, Johannes Erdfelt</div>
<div>> <<a href="mailto:johannes@erdfelt.com">johannes@erdfelt.com</a> <<a href="mailto:johannes@erdfelt.com>">mailto:johannes@erdfelt.com></a>> wrote:</div>
<div>> </div>
<div>> On Tue, Mar 11, 2014, Mike Wilson <<a href="mailto:geekinutah@gmail.com">geekinutah@gmail.com</a></div>
<div>> <<a href="mailto:geekinutah@gmail.com>">mailto:geekinutah@gmail.com></a>> wrote:</div>
<div>> > Undeleting things is an important use case in my opinion. We</div>
<div>> do this in our</div>
<div>> > environment on a regular basis. In that light I'm not sure</div>
<div>> that it would be</div>
<div>> > appropriate just to log the deletion and git rid of the row. I</div>
<div>> would like</div>
<div>> > to see it go to an archival table where it is easily restored.</div>
<div>> </div>
<div>> I'm curious, what are you undeleting and why?</div>
<div>> </div>
<div>> JE</div>
<div>> </div>
<div>> </div>
<div>> _______________________________________________</div>
<div>> OpenStack-dev mailing list</div>
<div>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a></div>
<div>> <<a href="mailto:OpenStack-dev@lists.openstack.org">mailto:OpenStack-dev@lists.openstack.org</a>></div>
<div>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></div>
<div>> </div>
<div>> </div>
<div>> </div>
<div>> _______________________________________________</div>
<div>> OpenStack-dev mailing list</div>
<div>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a></div>
<div>> <<a href="mailto:OpenStack-dev@lists.openstack.org">mailto:OpenStack-dev@lists.openstack.org</a>></div>
<div>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></div>
<div>> </div>
<div>> </div>
<div>> </div>
<div>> </div>
<div>> _______________________________________________</div>
<div>> OpenStack-dev mailing list</div>
<div>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a></div>
<div>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></div>
<div>> </div>
<div></div>
<div></div>
<div>_______________________________________________</div>
<div>OpenStack-dev mailing list</div>
<div><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a></div>
<div><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></div>
</blockquote>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>_______________________________________________</div>
<div>OpenStack-dev mailing list</div>
<div><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a></div>
<div><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></div>
<div><br>
</div>
</div>
</div>
</blockquote>
</span>
</body>
</html>