Open Stack

Hi Alexandr,

Have you read the DLO/SLO support in Swift ?
That might the thing you can leverage for managing chunks.
http://docs.openstack.org/developer/swift/overview_large_objects.html?highlight=large

Hugo

2016-03-11 16:40 GMT+08:00 Alexandr Porunov <alexandr.porunov at gmail.com>:

> Hello Hugo,
>
> Thank you for your response.
>
> It is example of my architecture of video upload:
> [image: Inline image 1]
> 1. User uploads video chunks to an upload server.
> 2. The upload server uploads each chunk to the temporary storage. (If
> connection breaks then the user will be able to recover it and continue
> download his video because his chunks already saved in the temporary
> storage)
> 3. After the video is fully saved, a upload server will send a task to a
> queue to transform the video.
> 4. A worker takes the task and transforms the video.
> 5. The worker uploads the video to the permanent storage.
> 6. The worker removes the video from temporary storage.
>
> Each chunk of the video is about 1 mb.
> Maximum size of the video is 8 GB.
> The upload server and the worker are written on Java.
> Permanent storage - OpenStack Swift.
> Transformation will starts only when full video will be uploaded.
>
> It will be great to use Swift as a temporary storage but I am not sure
> that I can manage chunks in Swift.
> If it is possible, please, give me any link to read about it.
>
> Best Regards, Alexandr
>
>
> On Fri, Mar 11, 2016 at 6:15 AM, Kuo Hugo <tonytkdk at gmail.com> wrote:
>
>> Hi Alexandr,
>>
>> The OpenStack Swift is a good choice as permanent storage pool for your
>> video case.
>> As for temporary storage media in your case, Swift could be an option.
>> But you need to consider several factors.
>>
>>    - How will user upload video to your site ?
>>    - How’s the speed do you need to store video chunks into temporary
>>    tier ?
>>    - What protocol is supported to store video by your application ?
>>    - Does your transformation server pull the data to it’s memory/local
>>    media to process ?
>>    - How many data will you have in your estimation ?
>>    - How’s the size of each video ingeneral ?
>>
>> Here’s a list of benefit by using Swift as both temporary & permanent
>> storage pool :
>>
>>    - User can upload video to Swift directly via HTTP. No need an
>>    additional server between your Swift pool and client.
>>    - Swift has built-in function for authentication/authorization.
>>    - If you’d like to move video cross temporary and permanent
>>    locations, it could be done in swift’s internal network.
>>    - The metadata can associate with object itself.
>>    - HTTP protocol gives high flexibility for various ways.
>>    - No video size limitation.
>>    - You can have 2 copies policy for temporary tier and 3 copies for
>>    permanent tier to get better data availability and durability.
>>
>> Here’s a list of potential weaknesses for temporary tier :
>>
>>    - If your transformation server needs to edit data online and change
>>    bits on each chunk, it’s not good for Swift.
>>
>> If the temporary tier doesn’t need too much space (lower than 3TB), you
>> can just use several high speed disks as a raid0 pool to have nice IOPs.
>>
>> Regard // Hugo
>> ​
>>
>> 2016-03-10 18:54 GMT+08:00 Alexandr Porunov <alexandr.porunov at gmail.com>:
>>
>>> Hello,
>>>
>>> I need a temporary storage for users videos.
>>> This is my scenario:
>>> 1. User uploads a video file chunk by chunk.
>>> 2. My application stores each chunk in the temporary storage.
>>> 3. When video loading is finished then my transformation server
>>> transforms this video and stores into permanent storage.
>>> 4. After video was saved in permanent storage it is deleted from
>>> temporary storage.
>>>
>>> Which storage is the best for read/write/delete only once and has the
>>> ability to save files chunk by chunk? Swift? Cinder? or something else?
>>>
>>> Sincerely, Alexandr
>>>
>>> _______________________________________________
>>> Mailing list:
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> Post to     : openstack at lists.openstack.org
>>> Unsubscribe :
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>
>>>
>>
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160314/6ed0dc8c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: upload_architecture.png
Type: image/png
Size: 49084 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160314/6ed0dc8c/attachment.png>