<div>
Do you have any performance tuning suggestions for others using this configuration?We have found guests [1] have various latency [2] accessing a ceph block device through KVM.</div><div><br></div><div>[1] 12.04 guest OS with 3.8 kernel w/ virtio on 12.04 hypervisor with 3.8 kernel</div><div>[2] In contrast to kernel RBD mounted on bare metal outside KVM</div><div><br></div><div>fio —filename=/dev/vdb —direct=1 —rw=randrw —ioengine=libaio —bs=$bs —rwmixread=$rw —iodepth=16 —numjobs=8 —runtime=120</div><div><br></div><div>bs=1k .. 4m</div><div>rw=0,100</div><div><br></div><div>John</div><div><br></div><div><span style="color: rgb(160, 160, 168);">On Thursday, March 20, 2014 at 11:27 PM, Tim Bell wrote:</span></div>
<blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;">
<span><div><div>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div>
<p style="margin: 0px;"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p style="margin: 0px;"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US">This is exactly how CERN uses ceph, for block storage with cinder and glance (using rbd). There is no need to have the additional
CephFS layer for these functions. We have a 3PB ceph pool (but not all is used for OpenStack)<o:p></o:p></span></p>
<p style="margin: 0px;"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p style="margin: 0px;"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US">Some details at
<a href="http://indico.cern.ch/event/300076/contribution/0/material/slides/0.pdf">
http://indico.cern.ch/event/300076/contribution/0/material/slides/0.pdf</a> and <a href="http://www.slideshare.net/Inktank_Ceph/cern-ceph-day-london-2013">http://www.slideshare.net/Inktank_Ceph/cern-ceph-day-london-2013</a><o:p></o:p></span></p>
<p style="margin: 0px;"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p style="margin: 0px;"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US">Tim<o:p></o:p></span></p>
<p style="margin: 0px;"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p style="margin: 0px;"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Abel Lopez [<a href="mailto:alopgeek@gmail.com">mailto:alopgeek@gmail.com</a>]
<br>
<b>Sent:</b> 21 March 2014 06:52<br>
<b>To:</b> gustavo panizzo <gfa><br>
<b>Cc:</b> <a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a><br>
<b>Subject:</b> Re: [Openstack-operators] Cinder with Ceph which option is good ?<o:p></o:p></span></p>
</div>
</div>
<p style="margin: 0px;"><o:p> </o:p></p>
<p style="margin: 0px;">While you're at it, you can also get a benefit from using ceph for glance. <br>
<br>
On Thursday, March 20, 2014, gustavo panizzo <gfa> <<a href="mailto:gfa@zumbi.com.ar">gfa@zumbi.com.ar</a>> wrote:<o:p></o:p></p><blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p style="margin: 0px;">use ceph as backend for cinder, it works fine :) doc is on inktank website<br>
<br>
i use cephfs for other stuff but i found it slow (fuse is to blame)<br>
--<br>
1AE0 322E B8F7 4717 BDEA BF1D 44BB 1BA7 9F6C 6333<br>
<br>
sent from mobile<o:p></o:p></p>
<div>
<p style="margin: 0px;">On March 20, 2014 9:24:08 AM GMT-03:00, Zeeshan Ali Shah <<a href="javascript:_e(%7B%7D,'cvml','zashah@kth.se');" target="_blank">zashah@kth.se</a>> wrote:<o:p></o:p></p><blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">HI, </span><o:p></o:p></p>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
</div>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">We are running ceph for S3 and Swift . <o:p></o:p></span></p>
</div>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
</div>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">For Cinder I was also thinking to use the ceph . Now there are few options . <o:p></o:p></span></p>
</div>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
</div>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">1. Should we totally drop cinder and use ceph rbd totally ? does ceph rbd integrate good with OS keystone and horizon<o:p></o:p></span></p>
</div>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">2. Can we some how connect cinder backend to use rbd ? <o:p></o:p></span></p>
</div>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">3. What if we use cephFS as backend for cinder , any performance issues ?<o:p></o:p></span></p>
</div>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><br clear="all">
<o:p></o:p></span></p>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
</div>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">any suggestion ?<o:p></o:p></span></p>
</div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">-- <o:p></o:p></span></p>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
</div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Regards<o:p></o:p></span></p>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
</div>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Zeeshan Ali Shah<o:p></o:p></span></p>
</div>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">System Administrator - PDC HPC<o:p></o:p></span></p>
</div>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">PhD researcher (IT security)<o:p></o:p></span></p>
</div>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Kungliga Tekniska Hogskolan<o:p></o:p></span></p>
</div>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><a href="tel:%2B46%208%20790%209115" target="_blank">+46 8 790 9115</a><o:p></o:p></span></p>
</div>
<div>
<p style="margin: 0px;"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><a href="http://www.pdc.kth.se/members/zashah" target="_blank">http://www.pdc.kth.se/members/zashah</a><o:p></o:p></span></p>
</div>
</div>
</div>
<pre style="text-align:center"><hr size="3" width="100%" align="center"></pre>
<pre><br>OpenStack-operators mailing list<br><a href="javascript:_e(%7B%7D,'cvml','OpenStack-operators@lists.openstack.org');" target="_blank">OpenStack-operators@lists.openstack.org</a><br><br><o:p></o:p></pre>
<pre><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><o:p></o:p></pre>
</blockquote></div>
</div>
</blockquote></div>
</div>
</div><div><div>_______________________________________________</div><div>OpenStack-operators mailing list</div><div><a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a></div><div><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a></div></div></div></span>
</blockquote>
<div>
<br>
</div>