[Openstack] Swift downloading only manifest of static large object (SLO)
Ben McCann
ben at benmccann.com
Fri Mar 14 00:40:04 UTC 2014
Thanks you guys sooo much for the help!!! I got this figured out. I was
using swift's --use-slo flag and apparently SoftLayer doesn't support that.
It be really great to add to the docs that SLO support is optional (and
turned off by default). I was definitely confused by that.
Thanks!
-Ben
On Thu, Mar 13, 2014 at 11:42 AM, Ben McCann <ben at benmccann.com> wrote:
> I managed to find out from SoftLayer that they're running 1.7.x
>
>
> On Thu, Mar 13, 2014 at 11:39 AM, Ben McCann <ben at benmccann.com> wrote:
>
>> The segments do appear to be there.
>>
>> I don't see any X-Object-Manifest header though or anything like that
>>
>> URL:
>> https://dal05.objectstorage.softlayer.net/v1/AUTH_c4fd4142-xxxx-xxxx-xxxx-xxxxxxxxxxxx/mongodb-backups/mongodb_backup_2014.03.09.09.00.03.tar.gz
>> Auth Token: AUTH_tkced5xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>> Account: AUTH_c4fd4142-xxxx-xxxx-xxxx-xxxxxxxxxxxx
>> Container: mongodb-backups
>> Object: mongodb_backup_2014.03.09.09.00.03.tar.gz
>> Content Type: application/x-tar
>> Content Length: 3895
>> Last Modified: Sun, 09 Mar 2014 09:58:09 GMT
>> ETag: 0a8ec37fb235ac99e16a726e1b17375b
>> Meta Mtime: 1394358254.226578
>> Accept-Ranges: bytes
>> X-Timestamp: 1394359089.40223
>> X-Trans-Id: tx912c6e12d63a409aa5362b6075c5f930
>>
>> Is this object likely to be stored wrong? Or maybe some misconfiguration
>> on their end like a load banalancer stripping out headers or something?
>>
>>
>> On Thu, Mar 13, 2014 at 11:30 AM, Clay Gerrard <clay.gerrard at gmail.com>wrote:
>>
>>> No, you can't really turn off DLO and it's been around for awhile (let's
>>> assume for now they support it).
>>>
>>> Can you verify the segments are showing up in the "container_segments"
>>> container? Can you verify the "X-Object-Manifest" header on the
>>> "container/file.tar.gz" object?
>>>
>>> "swift stat -v container file.tar.gz" should show it as just
>>> "Manifest", something like "container
>>> _segments/file.tar.gz/1394610073.993163/104857600/1000000/"
>>>
>>> Good Luck,
>>>
>>> -Clay
>>>
>>>
>>> On Thu, Mar 13, 2014 at 11:00 AM, Ben McCann <ben at benmccann.com> wrote:
>>>
>>>> Oh geeze, sorry, looking at those docs, it seems maybe I was mistaken
>>>> and am using Dynamic Large Objects because I uploaded the file using swift
>>>> -S. Is DLO also optional? Was it added at a certain version?
>>>>
>>>> Thanks so much for the help!
>>>> -Ben
>>>>
>>>>
>>>> On Thu, Mar 13, 2014 at 10:50 AM, Clay Gerrard <clay.gerrard at gmail.com>wrote:
>>>>
>>>>> You should check with your deployer to see what version of Swift they
>>>>> are running.
>>>>>
>>>>> the /info (capabilities) feature was added in 1.11 and I think Havana
>>>>> shipped with 1.10
>>>>>
>>>>> But I think SLO support has been around since 1.5 which should be in
>>>>> Havana, maybe even Grizzly. [1]
>>>>>
>>>>> Either way SLO support has always been optional (and off by default) -
>>>>> so your deployer may only support DLO. [2]
>>>>>
>>>>> 1.
>>>>> https://wiki.openstack.org/wiki/ReleaseNotes/Grizzly#OpenStack_Object_Storage_.28Swift.29
>>>>> 2.
>>>>> http://docs.openstack.org/developer/swift/overview_large_objects.html
>>>>>
>>>>>
>>>>> On Thu, Mar 13, 2014 at 10:31 AM, Ben McCann <ben at benmccann.com>wrote:
>>>>>
>>>>>> Oh, you're right, I did miss the -v option. Thanks!
>>>>>>
>>>>>> The curl command isn't working for me, however.
>>>>>>
>>>>>> StorageURL:
>>>>>> https://dal05.objectstorage.softlayer.net/v1/AUTH_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
>>>>>>
>>>>>> $ curl https://dal05.objectstorage.softlayer.net/info
>>>>>> Bad URL
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 13, 2014 at 2:59 AM, Kuo Hugo <tonytkdk at gmail.com> wrote:
>>>>>>
>>>>>>> Hi Ben,
>>>>>>>
>>>>>>> 1. What's the Swift CLI version are you using for now ?
>>>>>>> 2. The $>swift stat -v suppose to show you the storage_url. -v
>>>>>>> means verbose.
>>>>>>> 3. There's another choice via curl
>>>>>>>
>>>>>>> ~IRON_MAN$>curl -i https://$FQDN_of_AUTH_SERVER/auth/v1.0 -H
>>>>>>> "X-auth-user:johndoe" -H "x-auth-key:password_of_john"
>>>>>>> HTTP/1.1 200 OK
>>>>>>> X-Storage-Url: https://swift.example.com/v1/AUTH_johndoe
>>>>>>> X-Auth-Token: AUTH_tk0f4473fa1234567890
>>>>>>> Content-Type: text/html; charset=UTF-8
>>>>>>> Set-Cookie: X-Auth-Token=AUTH_tk0f4473fa1234567890; Path=/
>>>>>>> X-Storage-Token: AUTH_tk0f4473fa1234567890
>>>>>>> Content-Length: 0
>>>>>>> X-Trans-Id: txcfefd5396a0f4f4694707-0053217efd
>>>>>>>
>>>>>>>
>>>>>>> And then you can send a GET to https://swift.example.com/info
>>>>>>>
>>>>>>> $> curl https://cloud.swiftstack.com/info
>>>>>>>
>>>>>>>
>>>>>>> One more information here, the /info was available since Swift
>>>>>>> 1.11.0
>>>>>>> https://github.com/openstack/swift/blob/master/CHANGELOG#L120-L125
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2014-03-13 14:41 GMT+08:00 Ben McCann <ben at benmccann.com>:
>>>>>>>
>>>>>>> Thanks Samuel. That's super helpful. I didn't realize that SLO could
>>>>>>>> be turned on/off on the server. I thought it was merely a client-side
>>>>>>>> convention.
>>>>>>>>
>>>>>>>> swift stat doesn't return a StorageUrl for me. It just returns:
>>>>>>>> Account:
>>>>>>>> Containers:
>>>>>>>> Objects:
>>>>>>>> Bytes:
>>>>>>>> Meta Cdn-Id:
>>>>>>>> Meta Nas-Id:
>>>>>>>> X-Timestamp:
>>>>>>>> X-Trans-Id:
>>>>>>>> Accept-Ranges:
>>>>>>>>
>>>>>>>> I'll see if I can find out from someone at SoftLayer whether they
>>>>>>>> have an old version or have large objects turned off unless you can think
>>>>>>>> of any other ways I might be able to find out by querying the server.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Ben
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Mar 12, 2014 at 11:09 PM, Samuel Merritt <
>>>>>>>> sam at swiftstack.com> wrote:
>>>>>>>>
>>>>>>>>> On 3/12/14 9:14 PM, Ben McCann wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I'm attempting to download a large file which I have uploaded to
>>>>>>>>>> SoftLayer Object Storage using the swift tool. It's downloading
>>>>>>>>>> only the
>>>>>>>>>> manifest and is not downloading the segments. Any tips on
>>>>>>>>>> figuring out
>>>>>>>>>> what is going wrong?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It's entirely possible that the cluster in question lacks static
>>>>>>>>> large object support. Maybe they're running an old version of Swift
>>>>>>>>> (pre-1.8.0/Grizzly), or maybe they haven't enabled the static large object
>>>>>>>>> feature.
>>>>>>>>>
>>>>>>>>> What you'll need to do is probe the cluster to see if static large
>>>>>>>>> objects are enabled.
>>>>>>>>>
>>>>>>>>> The easy way is to politely ask the cluster with a GET request to
>>>>>>>>> /info. Run `swift stat -v`, then take the "StorageURL" field and change the
>>>>>>>>> path to be just "/info". Strip off any path that's already there.
>>>>>>>>>
>>>>>>>>> For example, if the output looks like
>>>>>>>>>
>>>>>>>>> StorageURL: http://192.168.22.2:8080/v1/AUTH_test
>>>>>>>>> Auth Token: AUTH_tk89454677b67348e09eefc1d6fe4dfa68
>>>>>>>>> Account: AUTH_test
>>>>>>>>> Containers: 1
>>>>>>>>> [...]
>>>>>>>>>
>>>>>>>>> then you'd simply run
>>>>>>>>>
>>>>>>>>> curl http://192.168.22.2:8080/info
>>>>>>>>>
>>>>>>>>> The response will be a JSON hash, so maybe run it through a JSON
>>>>>>>>> pretty-printer, e.g.
>>>>>>>>>
>>>>>>>>> curl http://192.168.22.2:8080/info | python -mjson.tool
>>>>>>>>>
>>>>>>>>> On my development machine, the response looks something like this
>>>>>>>>> (some items removed for brevity):
>>>>>>>>>
>>>>>>>>> {
>>>>>>>>> "container_quotas": {},
>>>>>>>>> "formpost": {},
>>>>>>>>> "slo": { <---- LOOK FOR THIS
>>>>>>>>> "max_manifest_segments": 1000,
>>>>>>>>> "max_manifest_size": 2097152,
>>>>>>>>> "min_segment_size": 71
>>>>>>>>> },
>>>>>>>>> "swift": {
>>>>>>>>> "account_listing_limit": 10000,
>>>>>>>>> "container_listing_limit": 10000,
>>>>>>>>> "max_account_name_length": 256,
>>>>>>>>> "max_container_name_length": 256,
>>>>>>>>> "max_file_size": 5368709122,
>>>>>>>>> "max_meta_count": 90,
>>>>>>>>> "max_meta_name_length": 128,
>>>>>>>>> "max_meta_value_length": 256,
>>>>>>>>> "max_object_name_length": 1024,
>>>>>>>>> "version": "1.12.0.106.ga08e813"
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> If you don't see SLO in there, then you're out of luck. Bug
>>>>>>>>> Softlayer and see if they'll turn it on.
>>>>>>>>>
>>>>>>>>> Now, it's also possible that you'll get a 412 response with the
>>>>>>>>> body "Bad URL" when you run the curl command above. That will tell you that
>>>>>>>>> the cluster is running 1.11.0 or earlier, but it won't tell you anything
>>>>>>>>> about SLO support. In that case, try uploading a normal file (cat picture
>>>>>>>>> or something) with the header "X-Static-Large-Object: yes". If that
>>>>>>>>> succeeds, then SLO is disabled; if SLO is enabled, then the PUT will fail.
>>>>>>>>>
>>>>>>>>> A more verbose description is available here:
>>>>>>>>> https://ask.openstack.org/en/question/7005/static-large-objects/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
>>>>>>>>> openstack
>>>>>>>>> Post to : openstack at lists.openstack.org
>>>>>>>>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
>>>>>>>>> openstack
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> about.me/benmccann
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Mailing list:
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>> Post to : openstack at lists.openstack.org
>>>>>>>> Unsubscribe :
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Mailing list:
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>> Post to : openstack at lists.openstack.org
>>>>>>> Unsubscribe :
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> about.me/benmccann
>>>>>>
>>>>>> _______________________________________________
>>>>>> Mailing list:
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>> Post to : openstack at lists.openstack.org
>>>>>> Unsubscribe :
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> about.me/benmccann
>>>>
>>>
>>>
>>
>>
>> --
>> about.me/benmccann
>>
>
>
>
> --
> about.me/benmccann
>
--
about.me/benmccann
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140313/ffd10ab6/attachment.html>
More information about the Openstack
mailing list