Open Stack

I've been working on some services that require the ability to exploit the
co-location of compute and data storage (via Swift) onto the same racks, and
I wanted to start a discussion to see what the best way of controlling the
physical placement of Swift replicas might be.

Quick overview:

Various services desire the ability to control the location of data placed
in Swift in order to minimize network saturation when moving data to
compute, or in the case of services like Hadoop, to ensure that compute can
be moved to wherever the data resides. Read/write latency can also be
minimized by allowing authorized services to place one or more replicas onto
the same rack (with other replicas being placed on separate racks). Fault
tolerance can also be enhanced by ensuring that some replica(s) are placed
onto separate racks. Breaking this down we come up with the following
potential requirements:

1. Swift should allow authorized services to place a given number of object
replicas onto a particular rack, and onto separate racks.
2. Swift should allow authorized services and administrators to learn which
racks an object resides on, along with endpoints.

While requirement 1 addresses the rack-local writing of objects, requirement
2 facilitates the rack-local reading of objects. Swift's middelware
currently offers a list endpoints capability which could allow services to
select an endpoint on the same rack to read an object from, but there
doesn't appear to be a comparable solution for authorized in cloud services.

Currently I'm not sure of the best way to approach this problem. While
storage policies might offer some solution, I'm interested to gather input
on how we might move forward on a solution that addresses these requirements
in as direct a way as possible. Please share your ideas and input. It's also
worth noting that a similar need exists for Nova which I'm addressing in a
separate message.

Cheers,
Jonathan


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150217/78aab922/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5517 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150217/78aab922/attachment.bin>