Hello , folks<div><br></div><div>Seems most of time consumed by the following code in obj/server.py</div><div><br></div><div><span lang="EN-US" style><font color="#ff0000">iter(lambda:
reader(self.network_chunk_size), '') </font></span></div><div><br></div><div>L591 - L605</div><div><span lang="EN-US" style><a href="https://github.com/openstack/swift/blob/master/swift/obj/server.py#L591">https://github.com/openstack/swift/blob/master/swift/obj/server.py#L591</a>
</span></div><div><br></div><div><br></div><div><br></div><div><p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:8pt"><font color="#333333">10323 Jul 10 14:34:42
object-server WTF: InitTime: 0.000183#012 SavingTime: 0.055627#012 OS-Write
0.000015 MetadataTime: 15.848296#012 UpdateContainerTime: 0.042656 X-Trans-ID :
tx7a2181d0e9444ef5a13f9f60f657288f</font></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:8pt"><font color="#333333">10324 Jul 10 14:34:42
object-server WTF: InitTime: 0.000248#012 SavingTime: 0.862101#012 OS-Write
0.000014 MetadataTime: 0.089192#012 UpdateContainerTime: 0.003802 X-Trans-ID :
tx37f8a2e958734083ba064f898e9fdcb2</font><font color="#7f7f7f"></font></span></p>
<p class="MsoNormal" style="background:white"><b><span lang="EN-US" style="font-size:9.0pt;color:#c00000">10325 Jul 10 14:34:42
object-server WTF: InitTime: 0.000379#012 SavingTime: 14.094034#012 OS-Write
0.000013 MetadataTime: 0.033566#012 UpdateContainerTime: 0.004655 X-Trans-ID :
tx9ef952731e5e463daa05a0c973907f32</span></b></p>
<p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:8pt"><font color="#333333">10326 Jul 10 14:34:42
object-server WTF: InitTime: 0.000310#012 SavingTime: 0.801216#012 OS-Write
0.000017 MetadataTime: 0.122491#012 UpdateContainerTime: 0.008453 X-Trans-ID :
tx6a5a0c634bf9439282ea4736e7ba7422</font></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:8pt"><font color="#333333">10327 Jul 10 14:34:42
object-server WTF: InitTime: 0.000176#012 SavingTime: 0.006937#012 OS-Write
0.000011 MetadataTime: 15.642381#012 UpdateContainerTime: 0.297634 X-Trans-ID :
tx1b0f4e03daef48d68cbfdc6c6e915a0b</font></span></p>
<span lang="EN-US" style="font-size:8pt;font-family:Calibri,sans-serif"><font color="#333333">10328 Jul 10 14:34:42 object-server WTF:
InitTime: 0.000268#012 SavingTime: 0.012993#012 OS-Write 0.000016 MetadataTime:
0.001211#012 UpdateContainerTime: 0.001846 X-Trans-ID :</font></span></div><div><span lang="EN-US" style="font-size:8pt;font-family:Calibri,sans-serif"><font color="#333333"><br></font></span></div><div><font color="#333333"><span lang="EN-US" style="font-size:8pt;font-family:Calibri,sans-serif"> </span><span style="font-size:10pt;background-color:white">As the above result , there’s
a request low down the average speed.</span></font></div><div><font color="#333333"><span style="font-size:10pt;background-color:white"><br></span></font></div><div><font color="#333333">What will cause </font><span style>iter(lambda: reader(self.network_chunk_size), '') </span><span style><font color="#333333">consumes lots of time?</font></span></div>
<div><span style><font color="#333333"><br></font></span></div><div><span style><font color="#333333">Too many files in XFS or anything else ? Could it possible be a bug ?</font></span></div><div><span style><font color="#333333"><br>
</font></span></div><div><span style><font color="#333333"><br></font></span></div><div><span style><font color="#333333">Thanks</font></span></div><p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:10.0pt;color:#7f7f7f"> </span></p>
<div><div class="gmail_quote">2012/7/4 Kuo Hugo <span dir="ltr"><<a href="mailto:tonytkdk@gmail.com" target="_blank">tonytkdk@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I found that updater and replicator could improve this issue.<div><br></div><div>In my original practice , for getting best performance , I only start main workers ( account-server , container-server , object-server) , And keep upload / download / delete objects over 1000000 times. </div>
<div><br></div><div>Issues:<br><br></div><div>1. XFS or Swift consumes lots of memory for some reason , does anyone know what's been cached(or buffered , cached usage is not too much though) in memory in this practice ? After running container/object replicator , those memory all released. I'm curious the contents in memory . Is that all about object's metadata or something else?</div>
<div><br></div><div>2. Plenty of 10s timeout in proxy-server's log . Due to timeout for getting final status of put object from storage node. </div><div>At beginning , object-workers complain about 3s timeout for updating container (async later). but there's not too much complains . As more and more put / get / delete operations , more and more timeout happend. </div>
<div>Seems that updater can improve this issue. </div><div>Does this behavior related to the number of data in pickle ?</div><div><br></div><div><br></div><div>Thanks</div><span class="HOEnZb"><font color="#888888"><div>
Hugo </div></font></span><div class="HOEnZb"><div class="h5"><div><br></div><div><br><div class="gmail_quote">
2012/7/2 Kuo Hugo <span dir="ltr"><<a href="mailto:tonytkdk@gmail.com" target="_blank">tonytkdk@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>Hi all , </div><div><br></div>I did several loading tests for swift in recent days. <div><br></div><div>I'm facing an issue ....... Hope you can share you consideration to me ... </div><div><br></div><div>My environment:</div>
<div>Swift-proxy with Tempauth in one server : 4 cores/32G rams </div><div><br></div><div>Swift-object + Swift-account + Swift-container in storage node * 3 , each for : 8 cores/32G rams 2TB SATA HDD * 7 </div><div>=====================================================================================</div>
<div>bench.conf :</div><div><br></div><div><div>[bench]</div><div>auth = <a href="http://172.168.1.1:8082/auth/v1.0" target="_blank">http://172.168.1.1:8082/auth/v1.0</a></div><div>user = admin:admin</div><div>key = admin</div>
<div>concurrency = 200</div>
<div>object_size = 4048</div><div>num_objects = 100000</div><div>num_gets = 100000</div><div>delete = yes</div></div><div>=====================================================================</div><div><br></div><div>After 70 rounds .....</div>
<div><br></div><div>PUT operations get lots of failures , but GET still works properly</div><div><b style="background-color:rgb(255,0,0)">ERROR log:</b></div><div><div>Jul 1 04:35:03 proxy-server ERROR with Object server <a href="http://192.168.100.103:36000/DISK6" target="_blank">192.168.100.103:36000/DISK6</a> re: <span style="background-color:rgb(51,204,0)">Trying to get final status of PUT</span> to /v1/AUTH_admin/af5862e653054f7b803d8cf1728412d2_6/24fc2f997bcc4986a86ac5ff992c4370: <font color="#cc0000">Timeout (10s)</font> (txn: txd60a2a729bae46be9b667d10063a319f) (client_ip: 172.168.1.2)</div>
<div>Jul 1 04:34:32 proxy-server ERROR with Object server <a href="http://192.168.100.103:36000/DISK2" target="_blank">192.168.100.103:36000/DISK2</a> re: <span style="background-color:rgb(51,204,0)">Expect: 100-continue</span> on /AUTH_admin/af5862e653054f7b803d8cf1728412d2_19/35993faa53b849a89f96efd732652e31:<font color="#ff0000"> Timeout (10s)</font></div>
</div><div><br></div><div><br></div><div>And kernel starts to report failed message as below</div><div><span style="background-color:rgb(204,0,0)"><b>kernel failed log:</b></span></div><div><span style="background-color:rgb(204,0,0)"><div>
76666 Jul 1 16:37:50 angryman-storage-03 kernel: [350840.020736] w83795 0-002f: Failed to read from register 0x03c, err -6</div><div> 76667 Jul 1 16:37:50 angryman-storage-03 kernel: [350840.052654] w83795 0-002f: Failed to read from register 0x015, err -6</div>
<div> 76668 Jul 1 16:37:50 angryman-storage-03 kernel: [350840.080613] w83795 0-002f: Failed to read from register 0x03c, err -6</div><div> 76669 Jul 1 16:37:50 angryman-storage-03 kernel: [350840.112583] w83795 0-002f: Failed to read from register 0x016, err -6</div>
<div> 76670 Jul 1 16:37:50 angryman-storage-03 kernel: [350840.144517] w83795 0-002f: Failed to read from register 0x03c, err -6</div><div> 76671 Jul 1 16:37:50 angryman-storage-03 kernel: [350840.176468] w83795 0-002f: Failed to read from register 0x017, err -6</div>
<div> 76672 Jul 1 16:37:50 angryman-storage-03 kernel: [350840.208455] w83795 0-002f: Failed to read from register 0x03c, err -6</div><div> 76673 Jul 1 16:37:51 angryman-storage-03 kernel: [350840.240410] w83795 0-002f: Failed to read from register 0x01b, err -6</div>
<div> 76674 Jul 1 16:37:51 angryman-storage-03 kernel: [350840.272Jul 1 17:05:28 angryman-storage-03 kernel: imklog 6.2.0, log source = /proc/kmsg started.</div><div><br></div><div>PUTs become slower and slower , from 1,200/s to 200/s ...</div>
<div><br></div><div>I'm not sure if this is a bug or that's the limitation of XFS. If it's an limit of XFS . How to improve it ?</div><div><br></div><div>An additional question is XFS seems consume lots of memory , does anyone know about the reason of this behavior?</div>
<div><br></div><div><br></div><div>Appreciate .......</div><span><font color="#888888"><div><span style="background-color:transparent"> </span></div></font></span></span></div><span><font color="#888888"><div>
<div><br></div>-- <br><div>+Hugo Kuo+</div><div><a href="mailto:tonytkdk@gmail.com" target="_blank">tonytkdk@gmail.com<br>
</a></div><div><a href="mailto:tonytkdk@gmail.com" target="_blank">+</a>886 935004793</div>
<br>
</div>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>+Hugo Kuo+</div><div><a href="mailto:tonytkdk@gmail.com" target="_blank">tonytkdk@gmail.com<br></a></div><div><a href="mailto:tonytkdk@gmail.com" target="_blank">+</a>886 935004793</div>
<br>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>+Hugo Kuo+</div><div><a href="mailto:tonytkdk@gmail.com" target="_blank">tonytkdk@gmail.com<br></a></div><div><a href="mailto:tonytkdk@gmail.com" target="_blank">+</a>886 935004793</div>
<br>
</div>