Open Stack

During the past month, Swift contributors have gathered in Austin,
Hong Kong, and online to discuss projects underway. There are some
major efforts underway, and I hope to lay them out and tie them
together here, so that we all know what the goals for the next six
months are.

The biggest feature set is storage policies. Storage policies will
give deployers and users incredible flexibility in how to manage their
data in the storage cluster. There are three basic parts to storage
policies.

First, given the global set of hardware available in a single Swift
cluster, choose which subset of hardware on which to store data. This
can be done by geography (e.g. US-East vs EU vs APAC vs global) or by
hardware properties (e.g. SATA vs SSDs). An obviously, the combination
can give a lot of flexibility.

Second, given the subset of hardware being used to store the data,
choose how to encode the data across that set of hardware. For
example, perhaps you have 2-replica, 3-replica, or erasure code
policies. Combining this with the hardware possibilities, you get e.g.
US-East reduced redundancy, global triple replicas, and EU erasure
coded.

Third, give the subset of hardware and how to store the data across
that hardware, control how Swift talks to a particular storage volume.
This may be optimized local file systems. This may be Gluster volumes.
This may be non-POSIX volumes like Seagate's new Kinetic drives.

We're well on our way to getting this set of work done. In Hong Kong
there was a demo of the current state of multi-ring support (for parts
one and two). We've also got a great start in refactoring the
interface between Swift and on-disk files and databases (for part
three).

But there is still a ton of work to do.

* we need to finalize the multi-ring work for the WSGI processes
* we need to ensure that large objects work with storage policies
* replication needs to be multi-ring aware
* auditing needs to be multi-ring aware
* updaters need to be multi-ring aware
* we need to write a multi-ring reconciler
* we need to merge all of this multi-ring stuff into master
* we need to finalize the DiskFile refactoring
* we need to properly refactor the DBBrokers
* we need the make sure the daemons can be extended to support different DiskFiles

The top-level blueprint for this is at
https://blueprints.launchpad.net/swift/+spec/storage-policies

Our target for the storage policy work is to support erasure coded
data within Swift. After the general storage policy work above is
done, we need to work on refactoring Swift's proxy server to make sure
its interaction with the storage nodes allows for differing storage
schemes.

I'm tremendously excited about the potential for storage policies. I
think it's the most significant development in Swift since the entire
project was open-sourced. Storage policies allow Swift to grow from
being the engine powering the world's largest storage clouds to a
storage platform enabling broader use cases by offering the
flexibility to very specifically match many different deployment
patterns.

Oh and if that's not enough to keep us all busy, there is other work
going on in the community, too. Some has been merged into Swift, and
some will stay in the ecosystem of tools for Swift, but they are all
important to a storage system. We've got an improved replication
platform merged into Swift, and it needs to be thoroughly tested and
polished. Once it's stable, we'll be able to build on it to really
improve Swift around MTTD and MTTR metrics. We're in the process of
refactoring metadata so that it is strongly separate into stuff that a
user can see and change and stuff the user can't see and change.

There is also some serious effort being put forth by a few companies
(including HP and IBM) to provide a way to add powerful metadata
searching into a Swift cluster. The ZeroVM team is interested in
extending Swift to better support large-scale data processing. The
Barbican team is looking in to providing good ways to offer encryption
for data stored in Swift. Others are looking in to how to grow
clusters (changing the partition power) and extend clusters
(transparently federating multiple Swift clusters).

Not all of this is going to be done by the OpenStack Icehouse release.
We cut stable releases of Swift fairly often, and these features will
roll out in those releases as they are done. My goal for Icehouse is
to see the storage policy work done and ready for production.

--John


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131120/bd4e66df/attachment.pgp>