<div dir="ltr">Hi Alexandr, <div><br></div><div>Have you read the DLO/SLO support in Swift ? </div><div>That might the thing you can leverage for managing chunks. </div><div><a href="http://docs.openstack.org/developer/swift/overview_large_objects.html?highlight=large">http://docs.openstack.org/developer/swift/overview_large_objects.html?highlight=large</a><br></div><div><br></div><div>Hugo</div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-03-11 16:40 GMT+08:00 Alexandr Porunov <span dir="ltr"><<a href="mailto:alexandr.porunov@gmail.com" target="_blank">alexandr.porunov@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div>Hello Hugo,</div><div><br></div><div>Thank you for your response.</div><div><br></div><div>It is example of my architecture of video upload:</div><div><img src="cid:ii_15364889ef1b5525" alt="Inline image 1" width="408" height="407"><br></div><div>1. User uploads video chunks to an upload server.</div><div>2. The upload server uploads each chunk to the temporary storage. (If connection breaks then the user will be able to recover it and continue download his video because his chunks already saved in the temporary storage)</div><div>3. After the video is fully saved, a upload server will send a task to a queue to transform the video.</div><div>4. A worker takes the task and transforms the video.</div><div>5. The worker uploads the video to the permanent storage.</div><div>6. The worker removes the video from temporary storage.</div><div><br></div><div>Each chunk of the video is about 1 mb.</div><div>Maximum size of the video is 8 GB.</div><div>The upload server and the worker are written on Java.</div><div>Permanent storage - OpenStack Swift.</div><div>Transformation will starts only when full video will be uploaded.</div><div><br></div><div>It will be great to use Swift as a temporary storage but I am not sure that I can manage chunks in Swift.</div><div>If it is possible, please, give me any link to read about it. </div><div><br></div><div>Best Regards, Alexandr</div><div><br></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 11, 2016 at 6:15 AM, Kuo Hugo <span dir="ltr"><<a href="mailto:tonytkdk@gmail.com" target="_blank">tonytkdk@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><p style="margin:0px 0px 1.2em!important">Hi Alexandr, </p>
<p style="margin:0px 0px 1.2em!important">The OpenStack Swift is a good choice as permanent storage pool for your video case.<br>As for temporary storage media in your case, Swift could be an option. But you need to consider several factors. </p>
<ul style="margin:1.2em 0px;padding-left:2em">
<li style="margin:0.5em 0px">How will user upload video to your site ?</li>
<li style="margin:0.5em 0px">How’s the speed do you need to store video chunks into temporary tier ? </li>
<li style="margin:0.5em 0px">What protocol is supported to store video by your application ? </li>
<li style="margin:0.5em 0px">Does your transformation server pull the data to it’s memory/local media to process ? </li>
<li style="margin:0.5em 0px">How many data will you have in your estimation ? </li>
<li style="margin:0.5em 0px">How’s the size of each video ingeneral ?</li>
</ul>
<p style="margin:0px 0px 1.2em!important">Here’s a list of benefit by using Swift as both temporary & permanent storage pool : </p>
<ul style="margin:1.2em 0px;padding-left:2em">
<li style="margin:0.5em 0px">User can upload video to Swift directly via HTTP. No need an additional server between your Swift pool and client. </li>
<li style="margin:0.5em 0px">Swift has built-in function for authentication/authorization. </li>
<li style="margin:0.5em 0px">If you’d like to move video cross temporary and permanent locations, it could be done in swift’s internal network. </li>
<li style="margin:0.5em 0px">The metadata can associate with object itself. </li>
<li style="margin:0.5em 0px">HTTP protocol gives high flexibility for various ways. </li>
<li style="margin:0.5em 0px">No video size limitation. </li>
<li style="margin:0.5em 0px">You can have 2 copies policy for temporary tier and 3 copies for permanent tier to get better data availability and durability. </li>
</ul>
<p style="margin:0px 0px 1.2em!important">Here’s a list of potential weaknesses for temporary tier : </p>
<ul style="margin:1.2em 0px;padding-left:2em">
<li style="margin:0.5em 0px">If your transformation server needs to edit data online and change bits on each chunk, it’s not good for Swift. </li>
</ul>
<p style="margin:0px 0px 1.2em!important">If the temporary tier doesn’t need too much space (lower than 3TB), you can just use several high speed disks as a raid0 pool to have nice IOPs. </p>
<p style="margin:0px 0px 1.2em!important">Regard // Hugo</p>
<div title="MDH:SGkgQWxleGFuZHIswqA8ZGl2Pjxicj48L2Rpdj48ZGl2PlRoZSBPcGVuU3RhY2sgU3dpZnQgaXMg
YSBnb29kIGNob2ljZSBhcyBwZXJtYW5lbnQgc3RvcmFnZSBwb29sIGZvciB5b3VyIHZpZGVvIGNh
c2UuJm5ic3A7PC9kaXY+PGRpdj5BcyBmb3IgdGVtcG9yYXJ5IHN0b3JhZ2UgbWVkaWEgaW4geW91
ciBjYXNlLCBTd2lmdCBjb3VsZCBiZSBhbiBvcHRpb24uIEJ1dCB5b3UgbmVlZCB0byBjb25zaWRl
ciBzZXZlcmFsIGZhY3RvcnMuJm5ic3A7PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj4qIEhvdyB3
aWxsIHVzZXIgdXBsb2FkIHZpZGVvIHRvIHlvdXIgc2l0ZSA/PC9kaXY+PGRpdj4qIEhvdydzIHRo
ZSBzcGVlZCBkbyB5b3UgbmVlZCB0byBzdG9yZSB2aWRlbyBjaHVua3MgaW50byB0ZW1wb3Jhcnkg
dGllciA/Jm5ic3A7PC9kaXY+PGRpdj4qIFdoYXQgcHJvdG9jb2wgaXMgc3VwcG9ydGVkIHRvIHN0
b3JlIHZpZGVvIGJ5IHlvdXIgYXBwbGljYXRpb24gPyZuYnNwOzwvZGl2PjxkaXY+KiBEb2VzIHlv
dXIgdHJhbnNmb3JtYXRpb24gc2VydmVyIHB1bGwgdGhlIGRhdGEgdG8gaXQncyBtZW1vcnkvbG9j
YWwgbWVkaWEgdG8gcHJvY2VzcyA/Jm5ic3A7PC9kaXY+PGRpdj4qIEhvdyBtYW55IGRhdGEgd2ls
bCB5b3UgaGF2ZSBpbiB5b3VyIGVzdGltYXRpb24gPyZuYnNwOzwvZGl2PjxkaXY+KiBIb3cncyB0
aGUgc2l6ZSBvZiBlYWNoIHZpZGVvIGluZ2VuZXJhbCA/PC9kaXY+PGRpdj4mbmJzcDs8L2Rpdj48
ZGl2PkhlcmUncyBhIGxpc3Qgb2YgYmVuZWZpdCBieSB1c2luZyBTd2lmdCBhcyBib3RoIHRlbXBv
cmFyeSAmYW1wOyBwZXJtYW5lbnQgc3RvcmFnZSBwb29sIDombmJzcDs8L2Rpdj48ZGl2Pjxicj48
L2Rpdj48ZGl2PiogVXNlciBjYW4gdXBsb2FkIHZpZGVvIHRvIFN3aWZ0IGRpcmVjdGx5IHZpYSBI
VFRQLiBObyBuZWVkIGFuIGFkZGl0aW9uYWwgc2VydmVyIGJldHdlZW4geW91ciBTd2lmdCBwb29s
IGFuZCBjbGllbnQuJm5ic3A7PC9kaXY+PGRpdj4qIFN3aWZ0IGhhcyBidWlsdC1pbiBmdW5jdGlv
biBmb3IgYXV0aGVudGljYXRpb24vYXV0aG9yaXphdGlvbi4mbmJzcDs8L2Rpdj48ZGl2PiogSWYg
eW91J2QgbGlrZSB0byBtb3ZlIHZpZGVvIGNyb3NzIHRlbXBvcmFyeSBhbmQgcGVybWFuZW50IGxv
Y2F0aW9ucywgaXQgY291bGQgYmUgZG9uZSBpbiBzd2lmdCdzIGludGVybmFsIG5ldHdvcmsuJm5i
c3A7PC9kaXY+PGRpdj4qIFRoZSBtZXRhZGF0YSBjYW4gYXNzb2NpYXRlIHdpdGggb2JqZWN0IGl0
c2VsZi4mbmJzcDs8L2Rpdj48ZGl2PiogSFRUUCBwcm90b2NvbCBnaXZlcyBoaWdoIGZsZXhpYmls
aXR5IGZvciB2YXJpb3VzIHdheXMuJm5ic3A7PC9kaXY+PGRpdj4qIE5vIHZpZGVvIHNpemUgbGlt
aXRhdGlvbi4mbmJzcDs8L2Rpdj48ZGl2PiogWW91IGNhbiBoYXZlIDIgY29waWVzIHBvbGljeSBm
b3IgdGVtcG9yYXJ5IHRpZXIgYW5kIDMgY29waWVzIGZvciBwZXJtYW5lbnQgdGllciB0byBnZXQg
YmV0dGVyIGRhdGEgYXZhaWxhYmlsaXR5IGFuZCBkdXJhYmlsaXR5LiZuYnNwOzwvZGl2PjxkaXY+
PGJyPjwvZGl2PjxkaXY+SGVyZSdzIGEgbGlzdCBvZiBwb3RlbnRpYWwmbmJzcDs8c3BhbiBzdHls
ZT0iY29sb3I6IHJnYig4NCwgODQsIDg0KTsgbGluZS1oZWlnaHQ6IDE4LjJweDsiPndlYWtuZXNz
ZXMgZm9yIHRlbXBvcmFyeSB0aWVyIDombmJzcDs8L3NwYW4+PC9kaXY+PGRpdj48YnI+PC9kaXY+
PGRpdj48Zm9udCBjb2xvcj0iIzU0NTQ1NCI+PHNwYW4gc3R5bGU9ImxpbmUtaGVpZ2h0OiAxOC4y
cHg7Ij4qIElmIHlvdXIgdHJhbnNmb3JtYXRpb24gc2VydmVyIG5lZWRzIHRvIGVkaXQgZGF0YSBv
bmxpbmUgYW5kIGNoYW5nZSBiaXRzIG9uIGVhY2ggY2h1bmssIGl0J3Mgbm90IGdvb2QgZm9yIFN3
aWZ0LiZuYnNwOzwvc3Bhbj48L2ZvbnQ+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj48YnI+PC9k
aXY+PGRpdj5JZiB0aGUgdGVtcG9yYXJ5IHRpZXIgZG9lc24ndCBuZWVkIHRvbyBtdWNoIHNwYWNl
IChsb3dlciB0aGFuIDNUQiksIHlvdSBjYW4ganVzdCB1c2Ugc2V2ZXJhbCBoaWdoIHNwZWVkIGRp
c2tzIGFzIGEgcmFpZDAgcG9vbCB0byBoYXZlIG5pY2UgSU9Qcy4mbmJzcDs8L2Rpdj48ZGl2Pjxi
cj48L2Rpdj48ZGl2PlJlZ2FyZCAvLyBIdWdvPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj48YnI+
PC9kaXY+PGRpdj48c3BhbiBzdHlsZT0iY29sb3I6IHJnYig4NCwgODQsIDg0KTsgbGluZS1oZWln
aHQ6IDE4LjJweDsiPjxicj48L3NwYW4+PC9kaXY+" style="min-height:0;width:0;max-height:0;max-width:0;overflow:hidden;font-size:0em;padding:0;margin:0"></div></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>2016-03-10 18:54 GMT+08:00 Alexandr Porunov <span dir="ltr"><<a href="mailto:alexandr.porunov@gmail.com" target="_blank">alexandr.porunov@gmail.com</a>></span>:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><div><span style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px">Hello,</span></div><div><span style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px"><br></span></div><span style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px">I need a temporary storage for users videos.</span><br style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px"><span style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px">This is my scenario:</span><br style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px"><span style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px">1. User uploads a video file chunk by chunk.</span><br style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px"><span style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px">2. My application stores each chunk in the temporary storage.</span><br style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px"><span style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px">3. When video loading is finished then my transformation server transforms this video and stores into permanent storage.</span><br style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px"><span style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px">4. After video was saved in permanent storage it is deleted from temporary storage.</span><br style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px"><br style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px"><span style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px">Which storage is the best for read/write/delete only once and has the ability to save files chunk by chunk? Swift? Cinder? or something else?</span><br style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px"><br style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px"><span style="color:rgb(75,75,75);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:14px;line-height:19.6px">Sincerely, Alexandr</span><br></div>
<br></div></div>_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></div><br></div>
</div></div><br>_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br></blockquote></div><br></div>