<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"><base href="x-msg://11/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi John, <div>please find my answers below - just to clarify, you are talking about the storage for the instances right? </div><div><br><div apple-content-edited="true"><br></div>
<br><div><div>Le 3 oct. 2013 à 11:04, John Ashford <<a href="mailto:logica111@hotmail.com">logica111@hotmail.com</a>> a écrit :</div><br class="Apple-interchange-newline"><blockquote type="cite"><div class="hmmessage" style="font-size: 12pt; font-family: Calibri; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div dir="ltr"><div style="margin: 0px; padding: 0px; ">1 – Glusterfs V Ceph<o:p></o:p></div><div style="margin: 0px; padding: 0px; ">Im reading a lot of different opinions about which of these is the best storage backend. My need is for a fully stable product that has fault tolerance built in. It needs to support maybe 400 low traffic web sites and a few very high traffic. I saw a Redhat diag suggesting throughput on a Gbit nic with 2 storage servers (Glusterfs) would be around 200Mbps. I can put quad nics in the 2 or 3 storage machines to give extra breathing room. Gluster is of course a mature product and has Redhat pushing it forward but some people complain of speed issues. Any real life experiences of throughput using Ceph? I know Ceph is new but it seems there is considerable weight behind its development so while some say its not production ready I wonder if anyone has the experience to refute/concur?</div></div></div></blockquote><br>Based on my experience, and other folks around' experience, we all noticed that GlusterFS doesn't seem to cope well as a shared backend for the instances . With my setup, around 9 compute nodes, all connected through Gbit, and Sata drives, I didn't had the 200 Mpbs/sec. What was their protocol? Measuring access speed, latency, and read-time from the shared directory <b>AND </b>from the instance itself should give you a good idea of what to expect with your infrastructure.</div><div>Regarding CephFS, it's getting better, but last time I checked, everytime I was running either iozone or vdbench on that, the server kept crashing, and I had to manually rebooting it. Next to that, the MDS are not active/active (not sure about the Cuttlefish release though), which means that solely one is accessed when an object is requested. I always found Ceph somehow *better* than GlusterFS, more realiable (based on my testing), even if it's more complicated to manage - you have a real overwiew of what's going on within your cluster. Everytime I had to debug GlusterFS, it was a real pain ; not to mention I had data corrupted and even missing sometime more than one. </div><div>Regarding the load, glusterfs deamons sucked out all the cpu constantly for</div><div> replicating objects.</div><div><br></div><div>So I say, deploy both, test, and pick one (iozone, vdbench, dd are three good tools to bench your storage)<br><blockquote type="cite"><div class="hmmessage" style="font-size: 12pt; font-family: Calibri; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div dir="ltr"><div style="margin: 0px; padding: 0px; "><o:p></o:p></div><div style="margin: 0px; padding: 0px; "><o:p> </o:p></div><div style="margin: 0px; padding: 0px; ">2 – vm instances on clustered storage</div></div></div></blockquote>You meant on local storage ?<br><blockquote type="cite"><div class="hmmessage" style="font-size: 12pt; font-family: Calibri; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div dir="ltr"><div style="margin: 0px; padding: 0px; "><o:p></o:p></div><div style="margin: 0px; padding: 0px; ">Im reading how if you run your vm instances on gluster/ceph you benefit from live migration and faster access times since disk access is usually to the local disk. I just need to clarify this – and this may fully expose my ignorance - but surely the instance runs on the compute node, not storage node so I don’t get how people are claiming its faster running vm instance on the storage cluster unless they are actually running compute on the storage cluster in which case you don’t have proper separation of compute/storage. Also, you would have the networking overhead unless running a compute node on storage cluster? What am I missing?!</div></div></div></blockquote>When you go with local storage, you lose some benefits, but gain others : </div><div><br></div><div><div><i>Pros for local :</i> </div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>- More performance</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>- Easier to manage (not any skill is necessary to manage the disk)</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>- Easier to repair (mkfs_adm check, fsck, etc…)</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>- Doesn't rely on network availability</div><div><br></div><div><i>Cons for local :</i><span class="Apple-tab-span" style="white-space: pre; "> </span></div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>- Live migration takes much much longer since you will need to copy the disk over</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>- Lose a node, lose everything : instances running on it, storage</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>- if you lose the disk, without backup you are screwed. WIth backup, it'll take much time to restore</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>- If you remove a file, you are screwed (lets say "terminate" instead of rebooting)</div><div><br></div><div><i>Pros for shared : </i></div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>- Easier to balance the workload in minutes</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>- More reliable since the file is copied in multiple locations</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>- If you lose a node or a disk, you can retrieve the file </div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>- MooseFS only feature : it has a trash, if you delete a file (let's say a terminated instance), you can browse the MooseFS trash (or "quanrantaine"). That's a sweet feature</div><div><br></div><div><i>Cons for shared :</i> </div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>- More costly in terms of configuration, network ports and knowledge</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>- Some corruption can happen if the algorithm doesn't cope well with split-brains situations</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>- Huge network-load (induced by the replicas, check, recycling, etc..) that requires a good architecting (dedicated network)</div><div><br></div><blockquote type="cite"><div class="hmmessage" style="font-size: 12pt; font-family: Calibri; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div dir="ltr"><div style="margin: 0px; padding: 0px; "><o:p></o:p></div><div style="margin: 0px; padding: 0px; "><o:p> </o:p></div><div style="margin: 0px; padding: 0px; "> Thanks<o:p></o:p></div><div style="margin: 0px; padding: 0px; ">John</div></div></div></blockquote><div><br></div><div>Regards,</div><div>Razique</div><blockquote type="cite"></blockquote></div><br></div></body></html>