Open Stack

On 13 Sep 2016, at 15:47, Michael Yoon wrote:

> Hello Alex and Clay,
>
> Pardon my jumping in here, we do something similar and have a similar use
> case (we chose 1MB segments for mobile, and 10MB segments for desktop).
> One question I have in comparing copy/bulk-delete vs keeping the manifest
> around is what is the overhead of the manifest within swift for retrieval
> (both DLO and SLO use case)?  My understanding is that in the DLO case, the
> listing is kept in the container, and that in the SLO case the manifest is
> a file in swift itself.  In both cases an extra write/read is necessary to
> store/lookup the manifest, and in the case of SLO it will in the
> containerdb ring (often SSD), and in the case of DLO it would be in the
> object ring (HDD).  Is my understanding correct?

You're mostly right. The DLO manifest becomes a container listing request with a given prefix. That makes DLO reads very expensive and prone to issues if the container hasn't been updated yet. SLO manifests are explicit and stored as an object in the system, without the container server being involved in any way in the read path (the same as "normal" objects).

So yes, there's the extra lookup that will delay time-to-first-byte for *LOs, but it's a lot more expensive for DLOs than for SLOs. In almost every single case, users should use SLOs.

There is a "speed limit" internally to Swift for large object segments. Swift can limit the number of segments per second that are returned to the client.[1] This protects the cluster from users who put thousands of 1-byte objects into a single manifest and then DOS the cluster. The default values are tuned to fully utilize a 1gbps network link with 100MB segments, IIRC.

In general, if you're using *LOs, you've go large objects, so the time-to-first-byte is a very small percentage of the overall transfer time.

Hope that info help.

[1] https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L669-L678



--John




>
> Thanks,
>
> Michael Yoon
>
> Chief Technology Officer
>
> m.718.791.8670
>
> michael.yoon at mimedia.com
>
> MiMedia, Inc.
>
> 32 Court St. Suite 1800
>
> Brooklyn, NY 11201
>
> Sign up! First 10 GB are on us. <http://mimed.io/r/23C03YR>
>
> On Tue, Sep 13, 2016 at 5:18 PM, Alexandr Porunov <
> alexandr.porunov at gmail.com> wrote:
>
>> Thank you Clay!
>>
>> I use ffmpeg with remote input and local output for first convertion. Here
>> is a simplified algorithm:
>> 1. Convert with h264 in the highest quality (till 4K) and save it in a
>> transformer server.
>> 2. Convert into lower quality locally. And get additional data (thumbnails
>> and so on).
>> 3. Upload all converted videos and additional data into Swift (as you
>> suggested with sub-segments and segments).
>>
>> Compressed videos then are available to the client throug the transmuxing
>> servers.
>>
>> Best regards,
>> Alexandr
>>
>> On Tue, Sep 13, 2016 at 8:55 PM, Clay Gerrard <clay.gerrard at gmail.com>
>> wrote:
>>
>>>
>>> On Tue, Sep 13, 2016 at 10:37 AM, Alexandr Porunov <
>>> alexandr.porunov at gmail.com> wrote:
>>>
>>>>
>>>> Correct me if I am wrong. Algorithm is the next:
>>>> 1. Upload 1MB sub-segments (up to 500 sub-segments per segment). After
>>>> they will be uploaded I will use "copy" request to create one 500MB
>>>> segment. After that I will delete useless 1MB segments.
>>>> 2. Upload up to 240 segments with 500MB and create a manifest for them.
>>>>
>>>>
>>> Is it correct? Will this algorithm be suitable for this situation?
>>>>
>>>>
>>> That sounds correct - I think the COPY/bulk-delete is optional - you
>>> could also just have the top level SLO point to the sub-segment SLO's
>>> manifest directly and leave all of the 1MB chunks stored in Swift as is -
>>> could potentially help smooth out the evenness of diskfullness overall.  I
>>> think 1MB is big enough to smooth out any connection overhead - but you
>>> could test with COPY consolidation of the backend subsegments to objects
>>> too.
>>>
>>> Whatever works.
>>>
>>> Regardless it'll still take awhile to download a 120GB object.
>>>
>>> Do you stream the uncompressed videos directly into the encoder as you
>>> upload the compressed format - or stage to disk and re-upload?  What about
>>> the playback of compressed videos - mp4 should http-pseudo stream directly
>>> to an html5 browser just fine!?
>>>
>>> Cool use case!  Good luck!
>>>
>>> -Clay
>>>
>>>
>>>
>>
>> _______________________________________________
>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
>> openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
>> openstack
>>
>>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160913/c4535f90/attachment.sig>