[openstack-dev] [Swift] erasure codes, digging deeper
me at not.mn
Wed Jul 17 22:42:24 UTC 2013
Last week we wrote a blog post about introducting erasure codes into Swift. Today, I'm happy to share more technical details around this feature.
We've posted an overview of our design and some of the tradeoffs in the feature at http://swiftstack.com/blog/2013/07/17/erasure-codes-with-openstack-swift-digging-deeper/.
I've also posted some slides with a high-level design at http://d.not.mn/EC_swift_proxy_design.pdf
Here's the summary of the erasure code design:
* Erasure codes (vs replicas) will be set on a per-container basis
* Data will be erasure coded inline in the proxy server
* API clients will control when data is replicated or erasure coded
So what's next? How do we get this done?
Of course, there's a lot of coding to be done. At a high level, and to tie everything together, I've updated Launchpad with several new blueprints. https://blueprints.launchpad.net/swift/+spec/swift-ec will keep track of the total work. More specifically, these are the areas that need to be worked on:
* Proxy server (https://blueprints.launchpad.net/swift/+spec/ec-proxy-work)
* EC Ring (https://blueprints.launchpad.net/swift/+spec/ec-ring)
* EC Reconstructor (https://blueprints.launchpad.net/swift/+spec/ec-reconstructor)
* EC Auditor (https://blueprints.launchpad.net/swift/+spec/ec-auditor)
* EC Stripe Auditor (https://blueprints.launchpad.net/swift/+spec/ec-stripe-auditor)
* EC library interface (https://blueprints.launchpad.net/swift/+spec/ec-library-interface)
Of course, as work progresses, there may be other areas too.
To facilitate the community dev work on this, I've done a couple of things.
First, there is now an upstream "feature/ec" branch for this work. The purpose of having a separate ec branch is because this is a long-running feature that does not have a lot of independent features (eg it doesn't make sense to merge an EC Reconstructor into Swift without the rest of the EC work). We will frequently merge master back into the ec branch so that we don't get too divergent. Here's how to start using this branch locally:
# go to the code and get the latest version
cd swift && git fetch --all
# checkout the upstream ec branch
git checkout origin/feature/ec
# create a local branch called ec to track the upstream ec branch
git branch ec2 origin/feature/ec && git checkout ec
# type, type, type
# this will push the review to gerrit for review (normal rules apply).
# on this ec branch, it will default to merge into the upstream feature/ec branch instead of master
Second, I've also set up a trello board to keep track and discuss designs at https://trello.com/board/swift-erasure-codes/51e0814d4ee9022d2b002a2c. Why trello? Mostly because Launchpad isn't sufficient to have discussions around the design, and gerrit only supports discussion around a particular piece of code.
Lastly, I would encourage anyone interested in this effort to participate in the #openstack-swift IRC channel on freenode and to attend the every-other-week swift meetings in #openstack-meeting (the next meeting is July 24 at 1900UTC).
I'm really looking forward to this feature, and I'm excited to see memebers of the OpenStack community coming together to implement it.
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 4082 bytes
Desc: not available
More information about the OpenStack-dev