<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="Content-Type" content="text/xhtml; charset=utf-8">
</head>
<body>
<div style="font-family:sans-serif"><div style="white-space:normal">
<h2 style="font-size:1.2em">Summary</h2>
<p dir="auto">Swift storage policies using ISA-L Vandermonde (<code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7">isa_l_rs_vand</code>) and<br>
having 5 or more parity bits will no longer be allowed. Swift's<br>
services will refuse to start unless these policies are deprecated.<br>
All existing data in these policies should be migrated to a different<br>
storage policy as soon as possible.</p>
<p dir="auto">Using ISA-L's Cauchy mode (<code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7">isa_l_rs_cauchy</code>) with 5 or more parity<br>
bits is safe (as is Cauchy mode with less than 5 parity bits). Using<br>
ISA-L's Vandermonde mode (<code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7">isa_l_rs_vand</code>) with less than 5 parity bits<br>
is safe.</p>
<p dir="auto">This change is expected in the next Swift release (2.15.0) and will<br>
be included in Pike.</p>
<h2 style="font-size:1.2em">Background</h2>
<p dir="auto">Late last year, we discovered that a particular config setting for<br>
erasure codes in Swift would expose a bug in one of the supported<br>
erasure coded libraries (Intel's ISA-L) and could result in data<br>
becoming corrupted. <strong>THIS DATA CORRUPTION BUG HAS BEEN FIXED</strong>, and<br>
it was included in liberasurecode 1.3.1. We also bumped the dependency<br>
version for liberasurecode in Swift to remove the immediate danger to<br>
Swift clusters.</p>
<p dir="auto">When we updated the liberasurecode version dependency, we also added a<br>
warning in Swift if we detected a storage policy using <code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7">isa_l_rs_vand</code><br>
with 5 or more parity bits. These changes were released in Swift 2.13.0<br>
(and as the OpenStack Ocata release).</p>
<p dir="auto">An example bad erasure code policy config in <code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7">/etc/swift.conf</code> is</p>
<pre style="background-color:#F7F7F7; border-radius:5px 5px 5px 5px; margin-left:15px; margin-right:15px; max-width:90vw; overflow-x:auto; padding:5px" bgcolor="#F7F7F7"><code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0" bgcolor="#F7F7F7">[storage-policy:2]
name = deepfreeze7-6
aliases = df76
policy_type = erasure_coding
ec_type = isa_l_rs_vand
ec_num_data_fragments = 7
ec_num_parity_fragments = 6
ec_object_segment_size = 1048576
</code></pre>
<p dir="auto">For deeper context and background, Swift's upstream erasure code docs are at<br>
<a href="https://docs.openstack.org/developer/swift/overview_erasure_code.html" style="color:#3983C4">https://docs.openstack.org/developer/swift/overview_erasure_code.html</a></p>
<h2 style="font-size:1.2em">What's about to happen?</h2>
<p dir="auto">After <a href="https://review.openstack.org/#/c/468105/" style="color:#3983C4">https://review.openstack.org/#/c/468105/</a> lands, Swift services<br>
won't start if you have an EC storage policy with <code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7">isa_l_rs_vand</code> and<br>
5 or more parity bits unless that policy is deprecated. No new containers<br>
with this policy will be able to be created. Existing objects will be<br>
readable and you can still write to containers previously created with<br>
this storage policy.</p>
<p dir="auto">This proposed patch is expected to be included in Swift's next 2.15.0<br>
release. The OpenStack Pike release will include either Swift 2.15.x<br>
or Swift 2.16.x.</p>
<h2 style="font-size:1.2em">Why now?</h2>
<p dir="auto">Although Swift and liberasurecode will no longer actively corrupt<br>
data, it's still possible that some failures will result in an<br>
inability to reconstruct missing erasure code fragments to restore<br>
full durability. Operators should immediately cease using<br>
<code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7">isa_l_rs_vand</code> with 5 or more parity bits, and migrate all data<br>
stored in a policy like that to a different storage policy. Since data<br>
movement takes time, this process should be started as soon as<br>
possible.</p>
<h2 style="font-size:1.2em">What do ops need to do right now?</h2>
<ol>
<li value="1">Ensure that you are using liberasurecode 1.4.0 or later.</li>
<li value="2">Identify any storage policies using <code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7">isa_l_rs_vand</code> with 5 or more
parity bits</li>
<li value="3">For each policy found, deprecate the storage policy.</li>
<li value="4">Operators should change the name of the bad policy to reflect its
deprecated state. After renaming this policy, an alias can be created on
the new policy that matches the name of the old policy. This will provide
continuity for client apps. See below for an example.</li>
<li value="5">If you need to keep an erasure code policy with the same data/parity
balance, create a new one using <code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7">isa_l_rs_cauchy</code> (this requires
liberasurecode 1.4.0 or later). Note that a new storage policy must have a
unique name.</li>
<li value="6">Begin migrating existing data from the deprecated policy to a different
storage policy. Depending on the amount of data stored, this may take a
long time. At this time, there are no upstream tools to facilitate this,
but the process is a matter of GET'ing the data from the existing container
(with the deprecated policy) and PUT'ing the data to the new container
(using the new policy).</li>
</ol>
<p dir="auto">One way to migrate the data to a new policy is to use Swift's<br>
container sync feature. Using container sync will preserve object<br>
metadata and timestamps. Another option is to write a tool to iterate<br>
over existing data and send COPY request to copy data to a new<br>
contaienr. The advantage of this second option is that it's cheap to<br>
get started and doesn't require changing anything on the server side.</p>
<p dir="auto">Note that when objects move from one container to another, though,<br>
their URLs will change.</p>
<p dir="auto"><strong>Caution</strong></p>
<p dir="auto">A deprecated policy cannot also be the default policy. Therefore if<br>
your default policy uses <code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7">isa_l_rs_vand</code> and 5 or more parity bits,<br>
you will need to configure a different default policy before<br>
deprecating the policy with the bad config. That may mean adding<br>
another storage policy to act as the default, or making another<br>
existing policy the default.</p>
<h2 style="font-size:1.2em">Examples</h2>
<p dir="auto">Good config after doing the above steps:</p>
<pre style="background-color:#F7F7F7; border-radius:5px 5px 5px 5px; margin-left:15px; margin-right:15px; max-width:90vw; overflow-x:auto; padding:5px" bgcolor="#F7F7F7"><code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0" bgcolor="#F7F7F7"># this policy is deprecated and replaced by storage policy 3
[storage-policy:2]
name = deepfreeze7-6-deprecated
policy_type = erasure_coding
ec_type = isa_l_rs_vand
ec_num_data_fragments = 7
ec_num_parity_fragments = 6
ec_object_segment_size = 1048576
deprecated = true
# this policy is the good replacement for the one above
[storage-policy:3]
name = deepfreeze7-6
aliases = df76
policy_type = erasure_coding
ec_type = isa_l_rs_cauchy
ec_num_data_fragments = 7
ec_num_parity_fragments = 6
ec_object_segment_size = 1048576
</code></pre>
<h2 style="font-size:1.2em">Need more help?</h2>
<p dir="auto">Please feel free to ask any questions either here on the mailing list<br>
or in #openstack-swift on freenode IRC.</p>
<p dir="auto">Thank you for placing your trust in us to store your data in Swift. We<br>
are all deeply saddened by this bug and the extra work it may cause<br>
operators.</p>
<p dir="auto">John Dickinson, Swift PTL<br>
The OpenStack Swift community</p>
</div>
</div>
</body>
</html>