[Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift

Leander Bessa Beernaert leanderbb at gmail.com
Tue Jan 15 11:09:44 UTC 2013


I've updated the keystoneclient to the lastest version available in GitHub
and I'm still not getting any speedups. I've switched to the tempauth
system and noticed an immediate increase in throughput (6.4 GB/day to 53.5
GB/day).

Since I'm time-restrained, I'll stick with the tempauth system for now.
I'll check back on the keystone token cache later.

Thanks for all the help.


On Tue, Jan 15, 2013 at 8:36 AM, Chmouel Boudjnah <chmouel at chmouel.com>wrote:

> as Chuck explain you usually would see that on the server, having said
> that if you use latest swiftclient from github you will be able to see the
> requests that swiftclient make via keystoneclient to keystone to get a
> token. If you go on the swift proxy server and only if you use a recent
> checked out version of swift you will be able to see the auth_token debug
> messages (the authentication middleware for validating a token to keystone)
> in your proxy-server log. I would advise to take a look at a devstack,
> running it with swift and see how it work on the swift proxy server console
> (in screen).
>
>
> On Mon, Jan 14, 2013 at 8:00 PM, Leander Bessa Beernaert <
> leanderbb at gmail.com> wrote:
>
>> Neither keystone nor swift proxy are producing any logs. I'm not sure
>> what to do :S
>>
>>
>> On Mon, Jan 14, 2013 at 6:50 PM, Chuck Thier <cthier at gmail.com> wrote:
>>
>>> You would have to look at the proxy log to see if a request is being
>>> made.  The results from the swift command line are just the calls that
>>> the client makes.  The server still haves to validate the token on
>>> every request.
>>>
>>> --
>>> Chuck
>>>
>>> On Mon, Jan 14, 2013 at 12:37 PM, Leander Bessa Beernaert
>>> <leanderbb at gmail.com> wrote:
>>> > Below is an output from Swift stat, since I don't see any requests to
>>> > keystone, I'm assuming that memcache is being used right?
>>> >
>>> > REQ: curl -i http://192.168.111.215:8080/v1/AUTH_[ID] -X HEAD -H
>>> > "X-Auth-Token: [TOKEN]"
>>> >
>>> > DEBUG:swiftclient:REQ: curl -i
>>> http://192.168.111.215:8080/v1/AUTH_[ID] -X
>>> > HEAD -H "X-Auth-Token: [TOKEN]"
>>> >
>>> > RESP STATUS: 204
>>> >
>>> > DEBUG:swiftclient:RESP STATUS: 204
>>> >
>>> >    Account: AUTH_[ID]
>>> > Containers: 44
>>> >    Objects: 4818
>>> >      Bytes: 112284450
>>> > Accept-Ranges: bytes
>>> > X-Timestamp: 1358184925.20885
>>> > X-Trans-Id: tx8cffb469c9c542be830db10a2b90d901
>>> >
>>> >
>>> >
>>> >
>>> > On Mon, Jan 14, 2013 at 6:31 PM, Dolph Mathews <
>>> dolph.mathews at gmail.com>
>>> > wrote:
>>> >>
>>> >> If memcache is being utilized by your keystone middleware, you should
>>> see
>>> >> keystone attaching to it on the first incoming request, e.g.:
>>> >>
>>> >>   keystoneclient.middleware.auth_token [INFO]: Using Keystone
>>> memcache for
>>> >> caching token
>>> >>
>>> >> You may also want to use auth_token from keystoneclient >= v0.2.0 if
>>> >> you're not already (instead of from keystone itself).
>>> >>
>>> >>
>>> >> -Dolph
>>> >>
>>> >>
>>> >> On Mon, Jan 14, 2013 at 11:43 AM, Leander Bessa Beernaert
>>> >> <leanderbb at gmail.com> wrote:
>>> >>>
>>> >>> Are you by any chance referring to this topic
>>> >>> https://lists.launchpad.net/openstack/msg08639.html regarding the
>>> keystone
>>> >>> token cache? If so I've already added the configuration line and
>>> have not
>>> >>> noticed any speedup :/
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Mon, Jan 14, 2013 at 5:19 PM, Leander Bessa Beernaert
>>> >>> <leanderbb at gmail.com> wrote:
>>> >>>>
>>> >>>> I'm using the ubuntu 12.04 packages of the folsom repository by the
>>> way.
>>> >>>>
>>> >>>>
>>> >>>> On Mon, Jan 14, 2013 at 5:18 PM, Chuck Thier <cthier at gmail.com>
>>> wrote:
>>> >>>>>
>>> >>>>> On Mon, Jan 14, 2013 at 11:03 AM, Leander Bessa Beernaert
>>> >>>>> <leanderbb at gmail.com> wrote:
>>> >>>>> > Also, I'm unable to run the swift-bench with keystone.
>>> >>>>> >
>>> >>>>>
>>> >>>>> Hrm... That was supposed to be fixed with this bug:
>>> >>>>> https://bugs.launchpad.net/swift/+bug/1011727
>>> >>>>>
>>> >>>>> My keystone dev instance isn't working at the moment, but I'll see
>>> if
>>> >>>>> I can get one of the team to take a look at it.
>>> >>>>>
>>> >>>>> --
>>> >>>>> Chuck
>>> >>>>
>>> >>>>
>>> >>>
>>> >>>
>>> >>> _______________________________________________
>>> >>> Mailing list: https://launchpad.net/~openstack
>>> >>> Post to     : openstack at lists.launchpad.net
>>> >>> Unsubscribe : https://launchpad.net/~openstack
>>> >>> More help   : https://help.launchpad.net/ListHelp
>>> >>>
>>> >>
>>> >
>>>
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack at lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130115/d157c1e6/attachment.html>


More information about the Openstack mailing list