Open Stack

Hello Clay,

Thank you for your suggestions.
I need to use Swift in several purposes.
1. As a temporary storage for uncompressed video scenes (y4m, mp4) up to
120GB. (After conversion they will be deleted).
2. As a persistent storage for compressed video scenes (mp4) up to 120GB.
Most of the compressed files will be much smaller (up to 15GB) but I need
to be able to store files up to 120GB in some rare situations.

Thanks for suggestions again. I will increase the size of segments to 1MB.

I haven't thought about two times segmentation. I think it will work.

Correct me if I am wrong. Algorithm is the next:
1. Upload 1MB sub-segments (up to 500 sub-segments per segment). After they
will be uploaded I will use "copy" request to create one 500MB segment.
After that I will delete useless 1MB segments.
2. Upload up to 240 segments with 500MB and create a manifest for them.

Is it correct? Will this algorithm be suitable for this situation?

Best regards,
Alexandr

On Tue, Sep 13, 2016 at 7:24 PM, Clay Gerrard <clay.gerrard at gmail.com>
wrote:

> 120GB is a rather largish file.  A standard 4TB drive is only gunna be
> able to hold a few tens of these.  My macbook could only hold two.
>
> 4KB at the bottom end of where you want to be in object storage - we're
> talking about the block device sector size at that point.
>
> The sweet spot for objects in Swift is probably in the 10MB-100MB range
> (2MB-200MB is fine).  Unless your primary platform is mobile there's a good
> chance the lower end of that range is reasonable target for resuming upload
> (oh, my last 500K of my 2MB upload crapped out - I'll just restart the
> other 1.5M, no bigz)
>
> But 120GB is a rather largish file.  The default SLO segment size is only
> like 1000 - but it'd be perfectly reasonable to bump that up an order of
> magnitude if you control the config for that cluster - then you could
> upload 20MB objects and call it done - but that doesn't give you much room
> to scale.
>
> I would suggest you consider breaking the 120GB upload into ~500MB
> sub-segments (each with ~500x1MB objects)  - then assemble the final SLO
> manifest from the sub-segments [1].
>
> Great to hear about people solving big storage problems with Swift - would
> you care to share anymore about your use-case?  Come find swifty folks in
> #openstack-swift on Freenode if you wanna chat!
>
> -Clay
>
> 1. final paragraph of http://docs.openstack.org/developer/swift/overview_
> large_objects.html#uploading-the-manifest
>
> On Tue, Sep 13, 2016 at 5:21 AM, Alexandr Porunov <
> alexandr.porunov at gmail.com> wrote:
>
>> Hello,
>>
>> Does somebody know how to minimize segments count of a file?
>> I have to implement an upload file service with the chunked upload
>> feature. If a client lose connection for some time we have to be able
>> continue to upload the file after connection is established (From the last
>> successfuly uploaded chunk). For it I will use "Static Large Objects" to
>> create small files each with 4 KBytes.
>>
>> The problem is that if we upload very big files then we will create too
>> many small 4 KBytes segments of the big file.
>>
>> My service have to be able to store files up to 120GB.
>> If we want to store a file with 120GB then we will have 31457280 segments
>> of the file in the object storage. It isn't nice because of additional cost
>> for metadate overhead of each file + big metadata file + decrease of
>> performance.
>>
>> It will be much better if we could combine all that segments in a bigger
>> segments each 1-2 GB. So, that after combination we would get maximum of
>> 60-120 segments of the file.
>>
>> I know that we can combine all segments into one by using "copy" command
>> to copy all segments into the new object but the problem is that if our
>> object will be more than 5 GB then we can't use "copy" command (because it
>> will show an error).
>>
>> Does somebody have any ideas how to develop chunk upload or minimize the
>> count of segments?
>>
>> Sincerely,
>> Alexandr
>>
>> _______________________________________________
>> Mailing list: http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160913/3c1c0e1c/attachment.html>