Open Stack

## Summary

Swift storage policies using ISA-L Vandermonde (`isa_l_rs_vand`) and
having 5 or more parity bits will no longer be allowed. Swift's
services will refuse to start unless these policies are deprecated.
All existing data in these policies should be migrated to a different
storage policy as soon as possible.

Using ISA-L's Cauchy mode (`isa_l_rs_cauchy`) with 5 or more parity
bits is safe (as is Cauchy mode with less than 5 parity bits). Using
ISA-L's Vandermonde mode (`isa_l_rs_vand`) with less than 5 parity bits
is safe.

This change is expected in the next Swift release (2.15.0) and will
be included in Pike.

## Background

Late last year, we discovered that a particular config setting for
erasure codes in Swift would expose a bug in one of the supported
erasure coded libraries (Intel's ISA-L) and could result in data
becoming corrupted. **THIS DATA CORRUPTION BUG HAS BEEN FIXED**, and
it was included in liberasurecode 1.3.1. We also bumped the dependency
version for liberasurecode in Swift to remove the immediate danger to
Swift clusters.

When we updated the liberasurecode version dependency, we also added a
warning in Swift if we detected a storage policy using `isa_l_rs_vand`
with 5 or more parity bits. These changes were released in Swift 2.13.0
(and as the OpenStack Ocata release).

An example bad erasure code policy config in `/etc/swift.conf` is

```ini
[storage-policy:2]
name = deepfreeze7-6
aliases = df76
policy_type = erasure_coding
ec_type = isa_l_rs_vand
ec_num_data_fragments = 7
ec_num_parity_fragments = 6
ec_object_segment_size = 1048576
```

For deeper context and background, Swift's upstream erasure code docs are at
<https://docs.openstack.org/developer/swift/overview_erasure_code.html>


## What's about to happen?

After <https://review.openstack.org/#/c/468105/> lands, Swift services
won't start if you have an EC storage policy with `isa_l_rs_vand` and
5 or more parity bits unless that policy is deprecated. No new containers
with this policy will be able to be created. Existing objects will be
readable and you can still write to containers previously created with
this storage policy.

This proposed patch is expected to be included in Swift's next 2.15.0
release. The OpenStack Pike release will include either Swift 2.15.x
or Swift 2.16.x.


## Why now?

Although Swift and liberasurecode will no longer actively corrupt
data, it's still possible that some failures will result in an
inability to reconstruct missing erasure code fragments to restore
full durability. Operators should immediately cease using
`isa_l_rs_vand` with 5 or more parity bits, and migrate all data
stored in a policy like that to a different storage policy. Since data
movement takes time, this process should be started as soon as
possible.


## What do ops need to do right now?

1. Ensure that you are using liberasurecode 1.4.0 or later.
2. Identify any storage policies using `isa_l_rs_vand` with 5 or more
   parity bits
3. For each policy found, deprecate the storage policy.
4. Operators should change the name of the bad policy to reflect its
   deprecated state. After renaming this policy, an alias can be created on
   the new policy that matches the name of the old policy. This will provide
   continuity for client apps. See below for an example.
5. If you need to keep an erasure code policy with the same data/parity
   balance, create a new one using `isa_l_rs_cauchy` (this requires
   liberasurecode 1.4.0 or later). Note that a new storage policy must have a
   unique name.
6. Begin migrating existing data from the deprecated policy to a different
   storage policy. Depending on the amount of data stored, this may take a
   long time. At this time, there are no upstream tools to facilitate this,
   but the process is a matter of GET'ing the data from the existing container
   (with the deprecated policy) and PUT'ing the data to the new container
   (using the new policy).

One way to migrate the data to a new policy is to use Swift's
container sync feature. Using container sync will preserve object
metadata and timestamps. Another option is to write a tool to iterate
over existing data and send COPY request to copy data to a new
contaienr. The advantage of this second option is that it's cheap to
get started and doesn't require changing anything on the server side.

Note that when objects move from one container to another, though,
their URLs will change.

**Caution**

A deprecated policy cannot also be the default policy. Therefore if
your default policy uses `isa_l_rs_vand` and 5 or more parity bits,
you will need to configure a different default policy before
deprecating the policy with the bad config. That may mean adding
another storage policy to act as the default, or making another
existing policy the default.


## Examples

Good config after doing the above steps:

```ini
# this policy is deprecated and replaced by storage policy 3
[storage-policy:2]
name = deepfreeze7-6-deprecated
policy_type = erasure_coding
ec_type = isa_l_rs_vand
ec_num_data_fragments = 7
ec_num_parity_fragments = 6
ec_object_segment_size = 1048576
deprecated = true

# this policy is the good replacement for the one above
[storage-policy:3]
name = deepfreeze7-6
aliases = df76
policy_type = erasure_coding
ec_type = isa_l_rs_cauchy
ec_num_data_fragments = 7
ec_num_parity_fragments = 6
ec_object_segment_size = 1048576
```

## Need more help?

Please feel free to ask any questions either here on the mailing list
or in #openstack-swift on freenode IRC.

Thank you for placing your trust in us to store your data in Swift. We
are all deeply saddened by this bug and the extra work it may cause
operators.

John Dickinson, Swift PTL
The OpenStack Swift community
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170612/c4289959/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170612/c4289959/attachment.sig>