Open Stack

Last week we wrote a blog post about introducting erasure codes into Swift. Today, I'm happy to share more technical details around this feature.

We've posted an overview of our design and some of the tradeoffs in the feature at http://swiftstack.com/blog/2013/07/17/erasure-codes-with-openstack-swift-digging-deeper/.

I've also posted some slides with a high-level design at http://d.not.mn/EC_swift_proxy_design.pdf

Here's the summary of the erasure code design:
    * Erasure codes (vs replicas) will be set on a per-container basis
    * Data will be erasure coded inline in the proxy server
    * API clients will control when data is replicated or erasure coded

So what's next? How do we get this done?

Of course, there's a lot of coding to be done. At a high level, and to tie everything together, I've updated Launchpad with several new blueprints. https://blueprints.launchpad.net/swift/+spec/swift-ec will keep track of the total work. More specifically, these are the areas that need to be worked on:
    * Proxy server (https://blueprints.launchpad.net/swift/+spec/ec-proxy-work)
    * EC Ring (https://blueprints.launchpad.net/swift/+spec/ec-ring)
    * EC Reconstructor (https://blueprints.launchpad.net/swift/+spec/ec-reconstructor)
    * EC Auditor (https://blueprints.launchpad.net/swift/+spec/ec-auditor)
    * EC Stripe Auditor (https://blueprints.launchpad.net/swift/+spec/ec-stripe-auditor)
    * EC library interface (https://blueprints.launchpad.net/swift/+spec/ec-library-interface)

Of course, as work progresses, there may be other areas too.

To facilitate the community dev work on this, I've done a couple of things.

First, there is now an upstream "feature/ec" branch for this work. The purpose of having a separate ec branch is because this is a long-running feature that does not have a lot of independent features (eg it doesn't make sense to merge an EC Reconstructor into Swift without the rest of the EC work). We will frequently merge master back into the ec branch so that we don't get too divergent. Here's how to start using this branch locally:
    
    # go to the code and get the latest version
    cd swift && git fetch --all
    # checkout the upstream ec branch
    git checkout origin/feature/ec
    # create a local branch called ec to track the upstream ec branch
    git branch ec2 origin/feature/ec && git checkout ec
    # type, type, type
    # this will push the review to gerrit for review (normal rules apply).
    # on this ec branch, it will default to merge into the upstream feature/ec branch instead of master
    git review

Second, I've also set up a trello board to keep track and discuss designs at https://trello.com/board/swift-erasure-codes/51e0814d4ee9022d2b002a2c. Why trello? Mostly because Launchpad isn't sufficient to have discussions around the design, and gerrit only supports discussion around a particular piece of code.

Lastly, I would encourage anyone interested in this effort to participate in the #openstack-swift IRC channel on freenode and to attend the every-other-week swift meetings in #openstack-meeting (the next meeting is July 24 at 1900UTC).

I'm really looking forward to this feature, and I'm excited to see memebers of the OpenStack community coming together to implement it.

--John
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4082 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130717/5d37f379/attachment.bin>