Open Stack

On 5/6/15 2:34 AM, Bala wrote:
> I am new to this list so please excuse me if I posted it in wrong list.
>
> We have a tape library which we would like to integrate with OpenStack
> Swift & Swift3 object storage service to provide S3 interface.
>
> The current file system we have for the library has been integrated with
> Swift storage service and manages changer robot & tapes.
>
> This works well for writing.
>
> However for reading, loading a tape takes longer when GET requests are
> received, in some cases over 5 minutes and this causes timeout error.
> Most of the data stored in these tapes are archival data. This get
> worsen when multiple GET requests received (muti-user) for objects which
> are stored in different tapes.
>
> Due to the longer read times, we are looking to provide Amazon S3
> Glacier like interface through Swift & Swift3 so that clients can issue
> a POST OBJECT RESTORE request and wait for the data to be moved to
> temporary store/cache.
>
> I have come across a similar request
>
> http://openstack-dev.openstack.narkive.com/kI72vk9l/ltfs-integration-with-openstack-swift-for-scenario-like-data-archival-as-a-service
>
> and understand the suggestions.
>
> We would like to provide S3 Glacier like interface than Swift Storage
> policies if we can.
>
> I would be great full if you could kindly advise
>
> 1. How hard is to change Swift & Swift3 code base to provide S3 Glacier
> like interface

It's not easy, that's for sure. Swift's API is all synchronous: issue a 
GET, receive the object; issue a PUT, create an object; et cetera. 
Glacier-style asynchronous retrieval is something completely new and 
different.

Some food for thought: where will you store pending retrieval requests? 
How will you ensure that retrieval requests survive disk and machine 
failures the way everything else in Swift does?

I'm not asking you to answer here (though you can if you want to, of 
course); I'm just trying to nudge your thoughts in the right direction.

> 2. Can this be done through Swift storage policies alone.

No. A storage policy determines where and how its objects are stored, 
which affects things like access times and storage cost. The API for 
accessing those objects does not change based on the storage policy.

> 3. Do we have to modify Swift Auditor service to do a tape based
> checking rather than object based.

You mean audit in order? Probably a good idea, otherwise your tapes will 
spend all day seeking.

> 4. Would Swift replication service cause frequent Tape change request.

I'd guess that it would, but nobody knows for sure. As far as I know, 
nobody has jammed tape-library support into Swift before. You're the 
first. Report back and let everyone know how it goes. :)