[Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift

Leander Bessa Beernaert leanderbb at gmail.com
Tue Jan 15 13:09:09 UTC 2013


Sorry, At the moment i'm unable to fetch the latest version of swift :s


On Tue, Jan 15, 2013 at 12:14 PM, Chmouel Boudjnah <chmouel at chmouel.com>wrote:

> any chances you can try with latest swift as well and set :
>
>  log_level = DEBUG
>
> in swift proxy-server.conf
>
> and what what the authtoken middleware is doing in the /var/log/syslog (or
> wherever syslog log on your distro).
>
> Chmouel.
>
>
>
> On Tue, Jan 15, 2013 at 12:09 PM, Leander Bessa Beernaert <
> leanderbb at gmail.com> wrote:
>
>> I've updated the keystoneclient to the lastest version available in
>> GitHub and I'm still not getting any speedups. I've switched to the
>> tempauth system and noticed an immediate increase in throughput (6.4 GB/day
>> to 53.5 GB/day).
>>
>> Since I'm time-restrained, I'll stick with the tempauth system for now.
>> I'll check back on the keystone token cache later.
>>
>> Thanks for all the help.
>>
>>
>> On Tue, Jan 15, 2013 at 8:36 AM, Chmouel Boudjnah <chmouel at chmouel.com>wrote:
>>
>>> as Chuck explain you usually would see that on the server, having said
>>> that if you use latest swiftclient from github you will be able to see the
>>> requests that swiftclient make via keystoneclient to keystone to get a
>>> token. If you go on the swift proxy server and only if you use a recent
>>> checked out version of swift you will be able to see the auth_token debug
>>> messages (the authentication middleware for validating a token to keystone)
>>> in your proxy-server log. I would advise to take a look at a devstack,
>>> running it with swift and see how it work on the swift proxy server console
>>> (in screen).
>>>
>>>
>>> On Mon, Jan 14, 2013 at 8:00 PM, Leander Bessa Beernaert <
>>> leanderbb at gmail.com> wrote:
>>>
>>>> Neither keystone nor swift proxy are producing any logs. I'm not sure
>>>> what to do :S
>>>>
>>>>
>>>> On Mon, Jan 14, 2013 at 6:50 PM, Chuck Thier <cthier at gmail.com> wrote:
>>>>
>>>>> You would have to look at the proxy log to see if a request is being
>>>>> made.  The results from the swift command line are just the calls that
>>>>> the client makes.  The server still haves to validate the token on
>>>>> every request.
>>>>>
>>>>> --
>>>>> Chuck
>>>>>
>>>>> On Mon, Jan 14, 2013 at 12:37 PM, Leander Bessa Beernaert
>>>>> <leanderbb at gmail.com> wrote:
>>>>> > Below is an output from Swift stat, since I don't see any requests to
>>>>> > keystone, I'm assuming that memcache is being used right?
>>>>> >
>>>>> > REQ: curl -i http://192.168.111.215:8080/v1/AUTH_[ID] -X HEAD -H
>>>>> > "X-Auth-Token: [TOKEN]"
>>>>> >
>>>>> > DEBUG:swiftclient:REQ: curl -i
>>>>> http://192.168.111.215:8080/v1/AUTH_[ID] -X
>>>>> > HEAD -H "X-Auth-Token: [TOKEN]"
>>>>> >
>>>>> > RESP STATUS: 204
>>>>> >
>>>>> > DEBUG:swiftclient:RESP STATUS: 204
>>>>> >
>>>>> >    Account: AUTH_[ID]
>>>>> > Containers: 44
>>>>> >    Objects: 4818
>>>>> >      Bytes: 112284450
>>>>> > Accept-Ranges: bytes
>>>>> > X-Timestamp: 1358184925.20885
>>>>> > X-Trans-Id: tx8cffb469c9c542be830db10a2b90d901
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Mon, Jan 14, 2013 at 6:31 PM, Dolph Mathews <
>>>>> dolph.mathews at gmail.com>
>>>>> > wrote:
>>>>> >>
>>>>> >> If memcache is being utilized by your keystone middleware, you
>>>>> should see
>>>>> >> keystone attaching to it on the first incoming request, e.g.:
>>>>> >>
>>>>> >>   keystoneclient.middleware.auth_token [INFO]: Using Keystone
>>>>> memcache for
>>>>> >> caching token
>>>>> >>
>>>>> >> You may also want to use auth_token from keystoneclient >= v0.2.0 if
>>>>> >> you're not already (instead of from keystone itself).
>>>>> >>
>>>>> >>
>>>>> >> -Dolph
>>>>> >>
>>>>> >>
>>>>> >> On Mon, Jan 14, 2013 at 11:43 AM, Leander Bessa Beernaert
>>>>> >> <leanderbb at gmail.com> wrote:
>>>>> >>>
>>>>> >>> Are you by any chance referring to this topic
>>>>> >>> https://lists.launchpad.net/openstack/msg08639.html regarding the
>>>>> keystone
>>>>> >>> token cache? If so I've already added the configuration line and
>>>>> have not
>>>>> >>> noticed any speedup :/
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> On Mon, Jan 14, 2013 at 5:19 PM, Leander Bessa Beernaert
>>>>> >>> <leanderbb at gmail.com> wrote:
>>>>> >>>>
>>>>> >>>> I'm using the ubuntu 12.04 packages of the folsom repository by
>>>>> the way.
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Mon, Jan 14, 2013 at 5:18 PM, Chuck Thier <cthier at gmail.com>
>>>>> wrote:
>>>>> >>>>>
>>>>> >>>>> On Mon, Jan 14, 2013 at 11:03 AM, Leander Bessa Beernaert
>>>>> >>>>> <leanderbb at gmail.com> wrote:
>>>>> >>>>> > Also, I'm unable to run the swift-bench with keystone.
>>>>> >>>>> >
>>>>> >>>>>
>>>>> >>>>> Hrm... That was supposed to be fixed with this bug:
>>>>> >>>>> https://bugs.launchpad.net/swift/+bug/1011727
>>>>> >>>>>
>>>>> >>>>> My keystone dev instance isn't working at the moment, but I'll
>>>>> see if
>>>>> >>>>> I can get one of the team to take a look at it.
>>>>> >>>>>
>>>>> >>>>> --
>>>>> >>>>> Chuck
>>>>> >>>>
>>>>> >>>>
>>>>> >>>
>>>>> >>>
>>>>> >>> _______________________________________________
>>>>> >>> Mailing list: https://launchpad.net/~openstack
>>>>> >>> Post to     : openstack at lists.launchpad.net
>>>>> >>> Unsubscribe : https://launchpad.net/~openstack
>>>>> >>> More help   : https://help.launchpad.net/ListHelp
>>>>> >>>
>>>>> >>
>>>>> >
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~openstack
>>>> Post to     : openstack at lists.launchpad.net
>>>> Unsubscribe : https://launchpad.net/~openstack
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130115/cc2cafff/attachment.html>


More information about the Openstack mailing list