<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;"><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">Hello All , </div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><br></div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">For a long time we are testing Ceph from Firefly to Kraken , tried to optimise many things which are very very common I guess like test tcmalloc version 2.1 , 2,4 , jemalloc , setting debugs 0/0 , op_tracker and such others and I believe with out hardware we almost reach to end of the road. </div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><br></div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">Some vendor tests mixed us a lot like samsung <a href="http://www.samsung.com/semiconductor/support/tools-utilities/All-Flash-Array-Reference-Design/downloads/Samsung_NVMe_SSDs_and_Red_Hat_Ceph_Storage_CS_20160712.pdf">http://www.samsung.com/semiconductor/support/tools-utilities/All-Flash-Array-Reference-Design/downloads/Samsung_NVMe_SSDs_and_Red_Hat_Ceph_Storage_CS_20160712.pdf</a> , DELL <a href="https://www.google.com.tr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwiA4Z28_pTSAhXCJZoKHSYVD0AQFggeMAA&url=http%3A%2F%2Fen.community.dell.com%2Ftechcenter%2Fcloud%2Fm%2Fdell_cloud_resources%2F20442913%2Fdownload&usg=AFQjCNGGADYZkbABD_GZ8YMct4E19KSAXA&sig2=YZCEHMq7tnXSpVydMDacIg">Dell PowerEdge R730xd Performance and Sizing Guide for Red Hat …</a> and from intel <a href="http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2015/20150813_S303E_Zhang.pdf">http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2015/20150813_S303E_Zhang.pdf</a></div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><br></div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">At the end using 3 replica (Actually most of vendors are testing with 2 but I believe that its very very wrong way to do because when some of failure happen you should wait 300 sec which is configurable but from blogs we understaood that sometimes OSDs can be down and up again because of that I believe very important to set related number but we do not want instances freeze ) with config below with 4K , random and fully write only .</div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><br></div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">I red a lot about OSD and OSD process eating huge CPU , yes it is and we are very well know that we couldn’t get total of iOPS capacity of each raw SSD drives.</div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><br></div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">My question is , can you pls share almost same or closer config or any config test or production results ? Key is write, not %70 of read % 30 write or full read things …</div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><br></div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">Hardware :</div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><br></div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">6 x Node </div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">Each Node Have : </div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">2 Socker CPU 1.8 GHZ each and total 16 core</div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">3 SSD + 12 HDD (SSDs are in journal mode 4 HDD to each SSD)</div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">Raid Cards Configured Raid 0</div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">We did not see any performance different with JBOD mode of raid card because of that continued with raid 0 </div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">Also raid card write back cache is used because its adding extra IOPS too ! </div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><br></div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">Achieved IOPS : 35 K (Single Client) </div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">We tested up to 10 Clients which ceph fairly share this usage like almost 4K for each </div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><br></div><div><font face="Calibri,sans-serif">Test Command : </font><span style="background-color: rgb(255, 255, 255);"><font face="Menlo" size="2">fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k —iodepth=256 --size=1G --numjobs=8 --readwrite=randwrite —group_reporting</font></span></div><div><br></div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><br></div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">Regards</div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">Vahric Muhtaryan</div></body></html>