<div dir="ltr"><div>Hello Clay,</div><div><br></div><div>Thank you for your suggestions.</div><div>I need to use Swift in several purposes.</div><div>1. As a temporary storage for uncompressed video scenes (y4m, mp4) up to 120GB. (After conversion they will be deleted).</div><div>2. As a persistent storage for compressed video scenes (mp4) up to 120GB. Most of the compressed files will be much smaller (up to 15GB) but I need to be able to store files up to 120GB in some rare situations.</div><div><br></div><div>Thanks for suggestions again. I will increase the size of segments to 1MB.</div><div><br></div><div>I haven't thought about two times segmentation. I think it will work.</div><div><br></div><div>Correct me if I am wrong. Algorithm is the next:</div><div>1. Upload 1MB sub-segments (up to 500 sub-segments per segment). After they will be uploaded I will use "copy" request to create one 500MB segment. After that I will delete useless 1MB segments.</div><div>2. Upload up to 240 segments with 500MB and create a manifest for them. </div><div><br></div><div>Is it correct? Will this algorithm be suitable for this situation?</div><div><br></div><div>Best regards,</div><div>Alexandr</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 13, 2016 at 7:24 PM, Clay Gerrard <span dir="ltr"><<a href="mailto:clay.gerrard@gmail.com" target="_blank">clay.gerrard@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">120GB is a rather largish file. A standard 4TB drive is only gunna be able to hold a few tens of these. My macbook could only hold two.<div><br></div><div>4KB at the bottom end of where you want to be in object storage - we're talking about the block device sector size at that point.</div><div><br></div><div>The sweet spot for objects in Swift is probably in the 10MB-100MB range (2MB-200MB is fine). Unless your primary platform is mobile there's a good chance the lower end of that range is reasonable target for resuming upload (oh, my last 500K of my 2MB upload crapped out - I'll just restart the other 1.5M, no bigz)</div><div><br></div><div>But 120GB is a rather largish file. The default SLO segment size is only like 1000 - but it'd be perfectly reasonable to bump that up an order of magnitude if you control the config for that cluster - then you could upload 20MB objects and call it done - but that doesn't give you much room to scale.</div><div><br></div><div>I would suggest you consider breaking the 120GB upload into ~500MB sub-segments (each with ~500x1MB objects) - then assemble the final SLO manifest from the sub-segments [1].</div><div><br></div><div>Great to hear about people solving big storage problems with Swift - would you care to share anymore about your use-case? Come find swifty folks in #openstack-swift on Freenode if you wanna chat!</div><div><br></div><div>-Clay</div><div><br></div><div>1. final paragraph of <a href="http://docs.openstack.org/developer/swift/overview_large_objects.html#uploading-the-manifest" target="_blank">http://docs.openstack.org/<wbr>developer/swift/overview_<wbr>large_objects.html#uploading-<wbr>the-manifest</a></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Tue, Sep 13, 2016 at 5:21 AM, Alexandr Porunov <span dir="ltr"><<a href="mailto:alexandr.porunov@gmail.com" target="_blank">alexandr.porunov@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr"><div>Hello,<br></div><div><br></div><div>Does somebody know how to minimize segments count of a file?</div><div>I have to implement an upload file service with the chunked upload feature. If a client lose connection for some time we have to be able continue to upload the file after connection is established (From the last successfuly uploaded chunk). For it I will use "Static Large Objects" to create small files each with 4 KBytes.</div><div><br></div><div>The problem is that if we upload very big files then we will create too many small 4 KBytes segments of the big file. </div><div><br></div><div>My service have to be able to store files up to 120GB.</div><div>If we want to store a file with 120GB then we will have 31457280 segments of the file in the object storage. It isn't nice because of additional cost for metadate overhead of each file + big metadata file + decrease of performance.</div><div><br></div><div>It will be much better if we could combine all that segments in a bigger segments each 1-2 GB. So, that after combination we would get maximum of 60-120 segments of the file.</div><div><br></div><div>I know that we can combine all segments into one by using "copy" command to copy all segments into the new object but the problem is that if our object will be more than 5 GB then we can't use "copy" command (because it will show an error).</div><div><br></div><div>Does somebody have any ideas how to develop chunk upload or minimize the count of segments?</div><div><br></div><div>Sincerely,</div><div>Alexandr</div></div>
<br></div></div>______________________________<wbr>_________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div>