<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"><meta name="Generator" content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang="EN-US" link="blue" vlink="purple"><div class="WordSection1"><p class="MsoNormal"> </p><p class="MsoNormal">Hi,</p><p class="MsoNormal"> </p><p class="MsoNormal">The multi_delete in swift3, perform sequential DELETE. In my 3 storage-node configuration to delete a 1000 objects, it took 30 second.</p><p class="MsoNormal"> </p><p class="MsoNormal">Following code change to create 100 thread pool to delete 1000 object took only 12 second. (This may even reduce if more storage nodes in picture).</p><p class="MsoNormal"> </p><p class="MsoNormal">If the following code change look fine, How can we formally (propose/review/commit) take this to the swift3 github code base ?</p><p class="MsoNormal"> </p><p class="MsoNormal"><b> </b></p><p class="MsoNormal"><b>Code diff:</b></p><p class="MsoNormal"> </p><p class="MsoNormal">diff --git a/swift3/swift-plugin-s3/swift3/controllers/multi_delete.py b/swift3/swift-plugin-s3/swift3/controllers/multi_delete.py</p><p class="MsoNormal">index 1bfde1d..5140529 100644</p><p class="MsoNormal">--- a/swift3/swift-plugin-s3/swift3/controllers/multi_delete.py</p><p class="MsoNormal">+++ b/swift3/swift-plugin-s3/swift3/controllers/multi_delete.py</p><p class="MsoNormal">@@ -21,9 +21,9 @@ from swift3.response import HTTPOk, S3NotImplemented, NoSuchKey, \</p><p class="MsoNormal"> from swift3.cfg import CONF</p><p class="MsoNormal"> from swift3.utils import LOGGER</p><p class="MsoNormal"> </p><p class="MsoNormal">-# Zadara-Begin</p><p class="MsoNormal">+from eventlet import GreenPool</p><p class="MsoNormal">+import copy</p><p class="MsoNormal"> MAX_MULTI_DELETE_BODY_SIZE = 262144</p><p class="MsoNormal">-# Zadara-End</p><p class="MsoNormal"> </p><p class="MsoNormal"> </p><p class="MsoNormal"> class MultiObjectDeleteController(Controller):</p><p class="MsoNormal">@@ -44,6 +44,24 @@ class MultiObjectDeleteController(Controller):</p><p class="MsoNormal"> </p><p class="MsoNormal"> return tostring(elem)</p><p class="MsoNormal"> </p><p class="MsoNormal">+ def async_delete(self, reqs, key, elem):</p><p class="MsoNormal">+ req = copy.copy(reqs)</p><p class="MsoNormal">+ req.object_name = key</p><p class="MsoNormal">+ try:</p><p class="MsoNormal">+ req.get_response(self.app, method='DELETE')</p><p class="MsoNormal">+ except NoSuchKey:</p><p class="MsoNormal">+ pass</p><p class="MsoNormal">+ except ErrorResponse as e:</p><p class="MsoNormal">+ error = SubElement(elem, 'Error')</p><p class="MsoNormal">+ SubElement(error, 'Key').text = key</p><p class="MsoNormal">+ SubElement(error, 'Code').text = e.__class__.__name__</p><p class="MsoNormal">+ SubElement(error, 'Message').text = e._msg</p><p class="MsoNormal">+ return</p><p class="MsoNormal">+</p><p class="MsoNormal">+ if not self.quiet:</p><p class="MsoNormal">+ deleted = SubElement(elem, 'Deleted')</p><p class="MsoNormal">+ SubElement(deleted, 'Key').text = key</p><p class="MsoNormal">+</p><p class="MsoNormal"> @bucket_operation</p><p class="MsoNormal"> def POST(self, req):</p><p class="MsoNormal"> """</p><p class="MsoNormal">@@ -90,27 +108,17 @@ class MultiObjectDeleteController(Controller):</p><p class="MsoNormal"> body = self._gen_error_body(error, elem, delete_list)</p><p class="MsoNormal"> return HTTPOk(body=body)</p><p class="MsoNormal"> </p><p class="MsoNormal">+ parallel_delete = 100</p><p class="MsoNormal">+ run_pool = GreenPool(size=parallel_delete)</p><p class="MsoNormal"> for key, version in delete_list:</p><p class="MsoNormal"> if version is not None:</p><p class="MsoNormal"> # TODO: delete the specific version of the object</p><p class="MsoNormal"> raise S3NotImplemented()</p><p class="MsoNormal"> </p><p class="MsoNormal">- req.object_name = key</p><p class="MsoNormal">-</p><p class="MsoNormal">- try:</p><p class="MsoNormal">- req.get_response(self.app, method='DELETE')</p><p class="MsoNormal">- except NoSuchKey:</p><p class="MsoNormal">- pass</p><p class="MsoNormal">- except ErrorResponse as e:</p><p class="MsoNormal">- error = SubElement(elem, 'Error')</p><p class="MsoNormal">- SubElement(error, 'Key').text = key</p><p class="MsoNormal">- SubElement(error, 'Code').text = e.__class__.__name__</p><p class="MsoNormal">- SubElement(error, 'Message').text = e._msg</p><p class="MsoNormal">- continue</p><p class="MsoNormal">-</p><p class="MsoNormal">- if not self.quiet:</p><p class="MsoNormal">- deleted = SubElement(elem, 'Deleted')</p><p class="MsoNormal">- SubElement(deleted, 'Key').text = key</p><p class="MsoNormal">+ run_pool.spawn(self.async_delete, req, key, elem)</p><p class="MsoNormal">+</p><p class="MsoNormal">+ # Wait for all the process to complete</p><p class="MsoNormal">+ run_pool.waitall()</p><p class="MsoNormal"> </p><p class="MsoNormal"> body = tostring(elem)</p><p class="MsoNormal"> </p></div></body></html>