Yes, that's why I was careful to clarify that I was talking about parity RAID. Performance should be fine otherwise.<div><br></div><div>--</div><div>Chuck<br><br><div class="gmail_quote">On Wed, Dec 19, 2012 at 8:26 PM, Hua ZZ Zhang <span dir="ltr"><<a href="mailto:zhuadl@cn.ibm.com" target="_blank">zhuadl@cn.ibm.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
<p><font face="sans-serif">Chuck, David,</font><br>
<br>
<font face="sans-serif">Thanks for your explanation and sharing.</font><br>
<font face="sans-serif">Since RAID 0 doesn't have parity or mirroring to provide low level redundancy which indicate there's no write penalty, it can improve overall performance for concurrent IO of multiple disks.</font><br>
<font face="sans-serif">I'm wondering if it make sense to use such kind of RAID without parity/mirroring to increase R/W performance and leave replication and distribution to higher level of Swift.</font><br>
<font face="sans-serif"><br>
</font><br>
<br>
<img width="16" height="16" src="cid:1__=C7BBF049DF95C6EA8f9e8a93df938@cn.ibm.com" border="0" alt="Inactive hide details for Chuck Thier ---2012-12-20 上午 12:35:58---Chuck Thier <cthier@gmail.com>"><font color="#424282" face="sans-serif">Chuck Thier ---2012-12-20 上午 12:35:58---Chuck Thier <<a href="mailto:cthier@gmail.com" target="_blank">cthier@gmail.com</a>></font><br>
<br>
</p><table width="100%" border="0" cellspacing="0" cellpadding="0">
<tbody><tr valign="top"><td style="background-image:url(cid:2__=C7BBF049DF95C6EA8f9e8a93df938@cn.ibm.com);background-repeat:no-repeat" width="40%">
<ul style="padding-left:72pt"><font size="1" face="sans-serif"><b>Chuck Thier <<a href="mailto:cthier@gmail.com" target="_blank">cthier@gmail.com</a>></b></font><font size="1" face="sans-serif"> </font><br>
<font size="1" face="sans-serif">Sent by: openstack-bounces+zhuadl=<a href="mailto:cn.ibm.com@lists.launchpad.net" target="_blank">cn.ibm.com@lists.launchpad.net</a></font>
<p><font size="1" face="sans-serif">2012-12-20 上午 12:33</font></p></ul>
</td><td width="60%">
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tbody><tr valign="top"><td width="1%"><img width="58" height="1" src="cid:3__=C7BBF049DF95C6EA8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<div align="right"><font size="1" face="sans-serif">To</font></div></td><td width="100%"><img width="1" height="1" src="cid:3__=C7BBF049DF95C6EA8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<ul style="padding-left:7pt"><font size="1" face="sans-serif">David Busby <<a href="mailto:d.busby@saiweb.co.uk" target="_blank">d.busby@saiweb.co.uk</a>>, </font></ul>
</td></tr>
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:3__=C7BBF049DF95C6EA8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<div align="right"><font size="1" face="sans-serif">cc</font></div></td><td width="100%"><img width="1" height="1" src="cid:3__=C7BBF049DF95C6EA8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<ul style="padding-left:7pt"><font size="1" face="sans-serif">"<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>" <<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>></font></ul>
</td></tr>
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:3__=C7BBF049DF95C6EA8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<div align="right"><font size="1" face="sans-serif">Subject</font></div></td><td width="100%"><img width="1" height="1" src="cid:3__=C7BBF049DF95C6EA8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<ul style="padding-left:7pt"><font size="1" face="sans-serif">Re: [Openstack] [swift] RAID Performance Issue</font></ul>
</td></tr>
</tbody></table>
<table border="0" cellspacing="0" cellpadding="0">
<tbody><tr valign="top"><td width="58"><img width="1" height="1" src="cid:3__=C7BBF049DF95C6EA8f9e8a93df938@cn.ibm.com" border="0" alt=""></td><td width="336"><img width="1" height="1" src="cid:3__=C7BBF049DF95C6EA8f9e8a93df938@cn.ibm.com" border="0" alt=""></td>
</tr>
</tbody></table>
</td></tr>
</tbody></table><div><div class="h5">
<br>
<tt><font>There are a couple of things to think about when using RAID (or more<br>
specifically parity RAID) with swift.<br>
<br>
The first has already been identified in that the workload for swift<br>
is very write heavy with small random IO, which is very bad for most<br>
parity RAID. In our testing, under heavy workloads, the overall RAID<br>
performance would degrade to be as slow as a single drive.<br>
<br>
It is very common for servers to have many hard drives (our first<br>
servers that we did testing with had 24 2T drives). During testing,<br>
RAID rebuilds were looking like they would take 2 weeks or so, which<br>
was not acceptable. While the array was in a degraded state, the<br>
overall performance of that box would suffer dramatically, which would<br>
have ripple effects across the rest of the cluster.<br>
<br>
We tried to make things work well with RAID 5 for quite a while as it<br>
would have made operations easier, and the code simpler since we<br>
wouldn't have had to handle many of the failure scenarios.<br>
<br>
Looking back, having to not rely on RAID has made swift a much more<br>
robust and fault tolerant platform.<br>
<br>
--<br>
Chuck<br>
<br>
On Wed, Dec 19, 2012 at 4:32 AM, David Busby <<a href="mailto:d.busby@saiweb.co.uk" target="_blank">d.busby@saiweb.co.uk</a>> wrote:<br>
> Hi Zang,<br>
><br>
> As JuanFra points out there's not much sense in using Swift on top of raid<br>
> as swift handel; extending on this RAID introduces a "write penalty"<br>
> (</font></tt><tt><font><a href="http://theithollow.com/2012/03/21/understanding-raid-penalty/" target="_blank">http://theithollow.com/2012/03/21/understanding-raid-penalty/</a></font></tt><tt><font>) this in turn<br>
> leads to performance issues, refer the link for write penalty's per<br>
> configuration.<br>
><br>
> As I recall (though this was from way back in October 2010) the suggested<br>
> method of deploying swift is onto standalone XFS drives, leaving swift to<br>
> handel the replication and distribution.<br>
><br>
><br>
> Cheers<br>
><br>
> David<br>
><br>
><br>
><br>
><br>
><br>
><br>
> On Wed, Dec 19, 2012 at 9:12 AM, JuanFra Rodriguez Cardoso<br>
> <<a href="mailto:juanfra.rodriguez.cardoso@gmail.com" target="_blank">juanfra.rodriguez.cardoso@gmail.com</a>> wrote:<br>
>><br>
>> Hi Zang:<br>
>><br>
>> Basically, it makes no sense to use Swift on top of RAID because Swift<br>
>> just delivers replication schema.<br>
>><br>
>> Regards,<br>
>> JuanFra.<br>
>><br>
>> 2012/12/19 Hua ZZ Zhang <<a href="mailto:zhuadl@cn.ibm.com" target="_blank">zhuadl@cn.ibm.com</a>><br>
>>><br>
>>> Hi,<br>
>>><br>
>>> I have read the admin document of Swift and find there's recommendation<br>
>>> of not using RAID 5 or 6 because swift performance degrades quickly with it.<br>
>>> Can anyone explain why this could happen? If the RAID is done by hardware<br>
>>> RAID controller, will the performance issue still exist?<br>
>>> Anyone can share such kind of experience of using RAID with Swift?<br>
>>> Appreciated for any suggestion from you.<br>
>>><br>
>>> -Zhang Hua<br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> Mailing list: </font></tt><tt><font><a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a></font></tt><tt><font><br>
>>> Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
>>> Unsubscribe : </font></tt><tt><font><a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a></font></tt><tt><font><br>
>>> More help : </font></tt><tt><font><a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a></font></tt><tt><font><br>
>>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> Mailing list: </font></tt><tt><font><a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a></font></tt><tt><font><br>
>> Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
>> Unsubscribe : </font></tt><tt><font><a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a></font></tt><tt><font><br>
>> More help : </font></tt><tt><font><a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a></font></tt><tt><font><br>
>><br>
><br>
><br>
> _______________________________________________<br>
> Mailing list: </font></tt><tt><font><a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a></font></tt><tt><font><br>
> Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
> Unsubscribe : </font></tt><tt><font><a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a></font></tt><tt><font><br>
> More help : </font></tt><tt><font><a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a></font></tt><tt><font><br>
><br>
<br>
_______________________________________________<br>
Mailing list: </font></tt><tt><font><a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a></font></tt><tt><font><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : </font></tt><tt><font><a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a></font></tt><tt><font><br>
More help : </font></tt><tt><font><a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a></font></tt><tt><font><br>
<br>
</font></tt><br>
</div></div><p></p></div></blockquote></div><br></div>