Open Stack

Hello Alex and Clay,

Pardon my jumping in here, we do something similar and have a similar use
case (we chose 1MB segments for mobile, and 10MB segments for desktop).
One question I have in comparing copy/bulk-delete vs keeping the manifest
around is what is the overhead of the manifest within swift for retrieval
(both DLO and SLO use case)?  My understanding is that in the DLO case, the
listing is kept in the container, and that in the SLO case the manifest is
a file in swift itself.  In both cases an extra write/read is necessary to
store/lookup the manifest, and in the case of SLO it will in the
containerdb ring (often SSD), and in the case of DLO it would be in the
object ring (HDD).  Is my understanding correct?

Thanks,

Michael Yoon

Chief Technology Officer

m.718.791.8670

michael.yoon at mimedia.com

MiMedia, Inc.

32 Court St. Suite 1800

Brooklyn, NY 11201

Sign up! First 10 GB are on us. <http://mimed.io/r/23C03YR>

On Tue, Sep 13, 2016 at 5:18 PM, Alexandr Porunov <
alexandr.porunov at gmail.com> wrote:

> Thank you Clay!
>
> I use ffmpeg with remote input and local output for first convertion. Here
> is a simplified algorithm:
> 1. Convert with h264 in the highest quality (till 4K) and save it in a
> transformer server.
> 2. Convert into lower quality locally. And get additional data (thumbnails
> and so on).
> 3. Upload all converted videos and additional data into Swift (as you
> suggested with sub-segments and segments).
>
> Compressed videos then are available to the client throug the transmuxing
> servers.
>
> Best regards,
> Alexandr
>
> On Tue, Sep 13, 2016 at 8:55 PM, Clay Gerrard <clay.gerrard at gmail.com>
> wrote:
>
>>
>> On Tue, Sep 13, 2016 at 10:37 AM, Alexandr Porunov <
>> alexandr.porunov at gmail.com> wrote:
>>
>>>
>>> Correct me if I am wrong. Algorithm is the next:
>>> 1. Upload 1MB sub-segments (up to 500 sub-segments per segment). After
>>> they will be uploaded I will use "copy" request to create one 500MB
>>> segment. After that I will delete useless 1MB segments.
>>> 2. Upload up to 240 segments with 500MB and create a manifest for them.
>>>
>>>
>> Is it correct? Will this algorithm be suitable for this situation?
>>>
>>>
>> That sounds correct - I think the COPY/bulk-delete is optional - you
>> could also just have the top level SLO point to the sub-segment SLO's
>> manifest directly and leave all of the 1MB chunks stored in Swift as is -
>> could potentially help smooth out the evenness of diskfullness overall.  I
>> think 1MB is big enough to smooth out any connection overhead - but you
>> could test with COPY consolidation of the backend subsegments to objects
>> too.
>>
>> Whatever works.
>>
>> Regardless it'll still take awhile to download a 120GB object.
>>
>> Do you stream the uncompressed videos directly into the encoder as you
>> upload the compressed format - or stage to disk and re-upload?  What about
>> the playback of compressed videos - mp4 should http-pseudo stream directly
>> to an html5 browser just fine!?
>>
>> Cool use case!  Good luck!
>>
>> -Clay
>>
>>
>>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160913/a30995e1/attachment.html>