<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">That helped -- using tempauth instead of keystone improved performance significantly on the openStack cluster.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">However performance is still slower compared to my Swift only setup where commands are sent directly to the swift node instead of going thru the controller
node in the openstack cluster. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">How does controller and swift node communications affect swift performance?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I’ve also noticed that all objects are stored on the 2<sup>nd</sup> swift node and not the 1<sup>st</sup> on the openstack cluster. I’m wondering if that could
also be a factor in slow performance.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Keystone:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">TOTAL<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> Count: 50 Average requests per second: 9.2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> min max avg std_dev 95%-ile Worst latency TX ID<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> First-byte latency: 0.067 - 2.513 0.390 ( 0.604) 1.948 (all obj sizes) txae75691d37d544b4ac0cfe3b8cba7f38<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> Last-byte latency: 0.067 - 3.337 0.430 ( 0.695) 1.997 (all obj sizes) txdcedb82227654b338daa85751f6d1232<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> First-byte latency: 0.070 - 2.513 0.542 ( 0.749) 2.255 ( tiny objs) txae75691d37d544b4ac0cfe3b8cba7f38<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> Last-byte latency: 0.070 - 2.514 0.468 ( 0.659) 1.997 ( tiny objs) txae75691d37d544b4ac0cfe3b8cba7f38<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> First-byte latency: 0.067 - 1.884 0.251 ( 0.382) 0.695 ( small objs) tx2ceec827f3304530b01a0d5993eea2e8<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> Last-byte latency: 0.067 - 3.337 0.385 ( 0.732) 1.884 ( small objs) txdcedb82227654b338daa85751f6d1232<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Tempauth:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-indent:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> Count: 50 Average requests per second: 65.7<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> min max avg std_dev 95%-ile Worst latency TX ID<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> First-byte latency: 0.006 - 0.073 0.014 ( 0.015) 0.055 (all obj sizes) tx69bf033a246645808b2c6a280e334f15<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> Last-byte latency: 0.006 - 0.248 0.047 ( 0.070) 0.198 (all obj sizes) txb8cf5dc0ce264eb08a1e05edbbf5a40f<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> First-byte latency: 0.006 - 0.073 0.017 ( 0.020) 0.072 ( tiny objs) tx69bf033a246645808b2c6a280e334f15<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> Last-byte latency: 0.006 - 0.248 0.053 ( 0.072) 0.195 ( tiny objs) txb8cf5dc0ce264eb08a1e05edbbf5a40f<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> First-byte latency: 0.006 - 0.026 0.010 ( 0.005) 0.026 ( small objs) tx65d1fd4b6ae049bb902442ac4c28ffe9<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> Last-byte latency: 0.006 - 0.218 0.040 ( 0.066) 0.198 ( small objs) txbfd6ebc74ed04068affd17c123572a44<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Swift Only:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">TOTAL<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> Count: 50 Average requests per second: 397.0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> min max avg std_dev 95%-ile Worst latency TX ID<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> First-byte latency: 0.003 - 0.007 0.005 ( 0.001) 0.006 (all obj sizes) None<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> Last-byte latency: 0.003 - 0.046 0.008 ( 0.009) 0.029 (all obj sizes) None<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> First-byte latency: 0.003 - 0.007 0.005 ( 0.001) 0.007 ( tiny objs) None<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> Last-byte latency: 0.003 - 0.046 0.008 ( 0.010) 0.027 ( tiny objs) None<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> First-byte latency: 0.004 - 0.006 0.005 ( 0.001) 0.006 ( small objs) None<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> Last-byte latency: 0.004 - 0.043 0.008 ( 0.009) 0.029 ( small objs) None<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Chmouel Boudjnah [mailto:chmouel@enovance.com]
<br>
<b>Sent:</b> Tuesday, July 09, 2013 8:22 AM<br>
<b>To:</b> OpenStack Development Mailing List<br>
<b>Subject:</b> Re: [openstack-dev] Swift debugging / performance - large latencies seen.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">On Tue, Jul 9, 2013 at 2:40 PM, Snider, Tim <<a href="mailto:Tim.Snider@netapp.com" target="_blank">Tim.Snider@netapp.com</a>> wrote:<o:p></o:p></p>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I have 2 openstack clusters running the Folsom release with multiple Swift nodes. I also have a small setup that is running only Swift with a single node. I’m noticing very large
Swift I/O latencies (seconds long) on the openstack clusters – ssbench output snippet is below. Performance is approximately identical on the openstack clusters. The Swift only cluster performs much better.
<o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
Keystone performance can be pretty awful unless you are using something else than the default WSGI container configuration (single process eventlet I think). I would suggest you try to run it under apache with multiple process.<br>
<br>
See the dicussion at last summit about Keystone performance here :<br>
<br>
<a href="https://etherpad.openstack.org/havana-keystone-performance">https://etherpad.openstack.org/havana-keystone-performance</a><br>
<br>
Chmouel.<o:p></o:p></p>
</div>
</body>
</html>