Open Stack

Hello,

Does somebody know how to minimize segments count of a file?
I have to implement an upload file service with the chunked upload feature.
If a client lose connection for some time we have to be able continue to
upload the file after connection is established (From the last successfuly
uploaded chunk). For it I will use "Static Large Objects" to create small
files each with 4 KBytes.

The problem is that if we upload very big files then we will create too
many small 4 KBytes segments of the big file.

My service have to be able to store files up to 120GB.
If we want to store a file with 120GB then we will have 31457280 segments
of the file in the object storage. It isn't nice because of additional cost
for metadate overhead of each file + big metadata file + decrease of
performance.

It will be much better if we could combine all that segments in a bigger
segments each 1-2 GB. So, that after combination we would get maximum of
60-120 segments of the file.

I know that we can combine all segments into one by using "copy" command to
copy all segments into the new object but the problem is that if our object
will be more than 5 GB then we can't use "copy" command (because it will
show an error).

Does somebody have any ideas how to develop chunk upload or minimize the
count of segments?

Sincerely,
Alexandr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160913/a4c51f3e/attachment.html>