Open Stack

Hello Hugo,

Thank you for your response.

It is example of my architecture of video upload:
[image: Inline image 1]
1. User uploads video chunks to an upload server.
2. The upload server uploads each chunk to the temporary storage. (If
connection breaks then the user will be able to recover it and continue
download his video because his chunks already saved in the temporary
storage)
3. After the video is fully saved, a upload server will send a task to a
queue to transform the video.
4. A worker takes the task and transforms the video.
5. The worker uploads the video to the permanent storage.
6. The worker removes the video from temporary storage.

Each chunk of the video is about 1 mb.
Maximum size of the video is 8 GB.
The upload server and the worker are written on Java.
Permanent storage - OpenStack Swift.
Transformation will starts only when full video will be uploaded.

It will be great to use Swift as a temporary storage but I am not sure that
I can manage chunks in Swift.
If it is possible, please, give me any link to read about it.

Best Regards, Alexandr


On Fri, Mar 11, 2016 at 6:15 AM, Kuo Hugo <tonytkdk at gmail.com> wrote:

> Hi Alexandr,
>
> The OpenStack Swift is a good choice as permanent storage pool for your
> video case.
> As for temporary storage media in your case, Swift could be an option. But
> you need to consider several factors.
>
>    - How will user upload video to your site ?
>    - How’s the speed do you need to store video chunks into temporary
>    tier ?
>    - What protocol is supported to store video by your application ?
>    - Does your transformation server pull the data to it’s memory/local
>    media to process ?
>    - How many data will you have in your estimation ?
>    - How’s the size of each video ingeneral ?
>
> Here’s a list of benefit by using Swift as both temporary & permanent
> storage pool :
>
>    - User can upload video to Swift directly via HTTP. No need an
>    additional server between your Swift pool and client.
>    - Swift has built-in function for authentication/authorization.
>    - If you’d like to move video cross temporary and permanent locations,
>    it could be done in swift’s internal network.
>    - The metadata can associate with object itself.
>    - HTTP protocol gives high flexibility for various ways.
>    - No video size limitation.
>    - You can have 2 copies policy for temporary tier and 3 copies for
>    permanent tier to get better data availability and durability.
>
> Here’s a list of potential weaknesses for temporary tier :
>
>    - If your transformation server needs to edit data online and change
>    bits on each chunk, it’s not good for Swift.
>
> If the temporary tier doesn’t need too much space (lower than 3TB), you
> can just use several high speed disks as a raid0 pool to have nice IOPs.
>
> Regard // Hugo
> ​
>
> 2016-03-10 18:54 GMT+08:00 Alexandr Porunov <alexandr.porunov at gmail.com>:
>
>> Hello,
>>
>> I need a temporary storage for users videos.
>> This is my scenario:
>> 1. User uploads a video file chunk by chunk.
>> 2. My application stores each chunk in the temporary storage.
>> 3. When video loading is finished then my transformation server
>> transforms this video and stores into permanent storage.
>> 4. After video was saved in permanent storage it is deleted from
>> temporary storage.
>>
>> Which storage is the best for read/write/delete only once and has the
>> ability to save files chunk by chunk? Swift? Cinder? or something else?
>>
>> Sincerely, Alexandr
>>
>> _______________________________________________
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160311/e20b5b11/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: upload_architecture.png
Type: image/png
Size: 49084 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160311/e20b5b11/attachment.png>