<div dir="ltr">Hi Robert, <div><br></div><div>Interesting post. </div><div><br></div><div>Appreciate ~</div></div><div class="gmail_extra"><br clear="all"><div><div dir="ltr"><div>+Hugo Kuo+</div><div><a href="mailto:hugo@swiftstack.com" target="_blank">hugo@swiftstack.com</a><br>
</div><div><a href="mailto:tonytkdk@gmail.com" target="_blank">tonytkdk@gmail.com<br></a></div><div>+886 935004793<br></div></div></div>
<br><br><div class="gmail_quote">2013/8/29 Klaus Schürmann <span dir="ltr"><<a href="mailto:klaus.schuermann@mediabeam.com" target="_blank">klaus.schuermann@mediabeam.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello Robert,<br>
<br>
very interesting experiences and thank you very much sharing them to us.<br>
<br>
Based on your experiences I made some tests with a 256 Byte inode size. I use Ubuntu with kernel 3.5.0 and its working properly.<br>
That should reduce the memory consumption and will perform better if I store much more objects.<br>
<br>
Deleting objects is a very strange behavior. We delete about 700.000 objects in the night and the load is rising 50 percent:<br>
<br>
<a href="http://www.schuermann.net/temp/delete.png" target="_blank">http://www.schuermann.net/temp/delete.png</a><br>
<br>
This is the network graph during deletion:<br>
<br>
<a href="http://www.schuermann.net/temp/delete-traffic.png" target="_blank">http://www.schuermann.net/temp/delete-traffic.png</a><br>
<br>
Normal?<br>
<br>
Regards,<br>
Klaus<br>
<br>
<br>
Von: Robert van Leeuwen [mailto:<a href="mailto:Robert.vanLeeuwen@spilgames.com">Robert.vanLeeuwen@spilgames.com</a>]<br>
Gesendet: Mittwoch, 28. August 2013 09:34<br>
An: <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
<div class="im">Betreff: Re: [Openstack] [SWIFT] PUTs and GETs getting slower<br>
<br>
</div><div class="im">Just a follow up on this thread because I've took some time to write up our experiences:<br>
<a href="http://engineering.spilgames.com/openstack-swift-lots-small-files/" target="_blank">http://engineering.spilgames.com/openstack-swift-lots-small-files/</a><br>
<br>
Klaus,<br>
<br>
Answering your question on initial sync times:<br>
Yes, we also see long initials syncs.<br>
For us it will take a few days for a new node to be synced.<br>
Usually it goes pretty quickly at first (30 MB/second) and the performance gradually degrades when the disks start filling up and the machines are running low on memory.<br>
We have about 6TB on a node to sync.<br>
<br>
Cheers,<br>
Robert van Leeuwen<br>
<br>
<br>
</div>________________________________________<br>
<div class="HOEnZb"><div class="h5">From: Klaus Schürmann [<a href="mailto:klaus.schuermann@mediabeam.com">klaus.schuermann@mediabeam.com</a>]<br>
Sent: Tuesday, August 20, 2013 9:04 AM<br>
To: Maximiliano Venesio; Robert van Leeuwen<br>
Cc: <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Subject: AW: [Openstack] [SWIFT] PUTs and GETs getting slower<br>
Hi,<br>
<br>
after adding additional disks and storing the account- and container-server on SSDs the performance is much better:<br>
<br>
Before:<br>
GETs average 620 ms<br>
PUTs average 1900 ms<br>
<br>
After:<br>
GETs average 280 ms<br>
PUTs average 1100 ms<br>
<br>
Only the rebalance process took days to sync all the data to the additional five disks (before each storage node had 3 disks). I used a concurrency of 4. One round to replicate all partitions took over 24 hours. After five days the replicate process takes only 300 seconds.<br>
Each additional disk has now 300 GB data stored. Is such duration normal to sync the data?<br>
<br>
Thanks<br>
Klaus<br>
<br>
<br>
Von: Maximiliano Venesio [mailto:<a href="mailto:maximiliano.venesio@mercadolibre.com">maximiliano.venesio@mercadolibre.com</a>]<br>
Gesendet: Donnerstag, 8. August 2013 17:26<br>
An: Robert van Leeuwen<br>
Cc: <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Betreff: Re: [Openstack] [SWIFT] PUTs and GETs getting slower<br>
<br>
Hi Robert, <br>
<br>
I was reading your post and is interesting because we have similar swift deployments and uses cases. <br>
We are storing millons of small images in our swift cluster, 32 Storage nodes w/12 - 2TB HDD + 2 SSD each one, and we are having an total average of 200k rpm in whole cluster.<br>
In terms of % of util. of our disks, we have an average of 50% of util in all our disks but we just are using a 15% of the total capacity of them.<br>
When I look at used inodes on our object nodes with "df -i" we hit about 17 million inodes per disk.<br>
<br>
So it seems a big number of inodes considering that we are using just a 15% of the total capacity. A different thing here is that we are using 512K of inode size and we have a big amount of memory . <br>
Also we always have one of our disks close to 100% of util, and this is caused by the object-auditor that scans all our disks continuously. <br>
<br>
So we was also thinking in the possibility to change the kind of disks that we are using, to use smaller and faster disks.<br>
Will be really util to know what kind of disks are you using in your old and new storage nodes, and compare that with our case.<br>
<br>
<br>
Cheers,<br>
Max<br>
<br>
<br>
<br>
<br>
<br>
</div></div><div class="HOEnZb"><div class="h5">Maximiliano Venesio <br>
#melicloud CloudBuilders<br>
Arias 3751, Piso 7 (C1430CRG) <br>
Ciudad de Buenos Aires - Argentina<br>
Cel: +549(11) 15-3770-1853<br>
Tel : <a href="tel:%2B54%2811%29%204640-8411" value="+541146408411">+54(11) 4640-8411</a><br>
<br>
On Tue, Aug 6, 2013 at 11:54 AM, Robert van Leeuwen <<a href="mailto:Robert.vanLeeuwen@spilgames.com">Robert.vanLeeuwen@spilgames.com</a>> wrote:<br>
Could you check your disk IO on the container /object nodes?<br>
<br>
We have quite a lot of files in swift and for comparison purposes I played a bit with COSbench to see where we hit the limits.<br>
We currently max out at about 200 - 300 put request/second and the bottleneck is the disk IO on the object nodes<br>
Our account / container nodes are on SSD's and are not a limiting factor.<br>
<br>
You can look for IO bottlenecks with e.g. "iostat -x 10" (this will refresh the view every 10 seconds.)<br>
During the benchmark is see some of the disks are hitting 100% utilization.<br>
That it is hitting the IO limits with just 200 puts a second has to do with the number of files on the disks.<br>
When I look at used inodes on our object nodes with "df -i" we hit about 60 million inodes per disk.<br>
(a significant part of that are actually directories I calculated about 30 million files based on the number of files in swift)<br>
We use flashcache in front of those disks and it is still REALLY slow, just doing a "ls" can take up to 30 seconds.<br>
Probably adding lots of memory should help caching the inodes in memory but that is quite challenging:<br>
I am not sure how big a directory is in the xfs inode tree but just the files:<br>
30 million x 1k inodes = 30GB<br>
And that is just one disk :)<br>
<br>
We still use the old recommended inode size of 1k and the default of 256 can be used now with recent kernels:<br>
<a href="https://lists.launchpad.net/openstack/msg24784.html" target="_blank">https://lists.launchpad.net/openstack/msg24784.html</a><br>
<br>
So sometime ago we decided to go for nodes with more,smaller & faster disks with more memory.<br>
Those machines are not even close to their limits however we still have more "old" nodes<br>
so performance is limited by those machines.<br>
At this moment it is sufficient for our use case but I am pretty confident we would be able to<br>
significantly improve performance by adding more of those machines and doing some re-balancing of the load.<br>
<br>
Cheers,<br>
Robert van Leeuwen<br>
_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br>
<br>
_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
</div></div></blockquote></div><br></div>