[Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift

Chmouel Boudjnah chmouel at chmouel.com
Tue Jan 15 12:14:30 UTC 2013


any chances you can try with latest swift as well and set :

 log_level = DEBUG

in swift proxy-server.conf

and what what the authtoken middleware is doing in the /var/log/syslog (or
wherever syslog log on your distro).

Chmouel.


On Tue, Jan 15, 2013 at 12:09 PM, Leander Bessa Beernaert <
leanderbb at gmail.com> wrote:

> I've updated the keystoneclient to the lastest version available in GitHub
> and I'm still not getting any speedups. I've switched to the tempauth
> system and noticed an immediate increase in throughput (6.4 GB/day to 53.5
> GB/day).
>
> Since I'm time-restrained, I'll stick with the tempauth system for now.
> I'll check back on the keystone token cache later.
>
> Thanks for all the help.
>
>
> On Tue, Jan 15, 2013 at 8:36 AM, Chmouel Boudjnah <chmouel at chmouel.com>wrote:
>
>> as Chuck explain you usually would see that on the server, having said
>> that if you use latest swiftclient from github you will be able to see the
>> requests that swiftclient make via keystoneclient to keystone to get a
>> token. If you go on the swift proxy server and only if you use a recent
>> checked out version of swift you will be able to see the auth_token debug
>> messages (the authentication middleware for validating a token to keystone)
>> in your proxy-server log. I would advise to take a look at a devstack,
>> running it with swift and see how it work on the swift proxy server console
>> (in screen).
>>
>>
>> On Mon, Jan 14, 2013 at 8:00 PM, Leander Bessa Beernaert <
>> leanderbb at gmail.com> wrote:
>>
>>> Neither keystone nor swift proxy are producing any logs. I'm not sure
>>> what to do :S
>>>
>>>
>>> On Mon, Jan 14, 2013 at 6:50 PM, Chuck Thier <cthier at gmail.com> wrote:
>>>
>>>> You would have to look at the proxy log to see if a request is being
>>>> made.  The results from the swift command line are just the calls that
>>>> the client makes.  The server still haves to validate the token on
>>>> every request.
>>>>
>>>> --
>>>> Chuck
>>>>
>>>> On Mon, Jan 14, 2013 at 12:37 PM, Leander Bessa Beernaert
>>>> <leanderbb at gmail.com> wrote:
>>>> > Below is an output from Swift stat, since I don't see any requests to
>>>> > keystone, I'm assuming that memcache is being used right?
>>>> >
>>>> > REQ: curl -i http://192.168.111.215:8080/v1/AUTH_[ID] -X HEAD -H
>>>> > "X-Auth-Token: [TOKEN]"
>>>> >
>>>> > DEBUG:swiftclient:REQ: curl -i
>>>> http://192.168.111.215:8080/v1/AUTH_[ID] -X
>>>> > HEAD -H "X-Auth-Token: [TOKEN]"
>>>> >
>>>> > RESP STATUS: 204
>>>> >
>>>> > DEBUG:swiftclient:RESP STATUS: 204
>>>> >
>>>> >    Account: AUTH_[ID]
>>>> > Containers: 44
>>>> >    Objects: 4818
>>>> >      Bytes: 112284450
>>>> > Accept-Ranges: bytes
>>>> > X-Timestamp: 1358184925.20885
>>>> > X-Trans-Id: tx8cffb469c9c542be830db10a2b90d901
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Mon, Jan 14, 2013 at 6:31 PM, Dolph Mathews <
>>>> dolph.mathews at gmail.com>
>>>> > wrote:
>>>> >>
>>>> >> If memcache is being utilized by your keystone middleware, you
>>>> should see
>>>> >> keystone attaching to it on the first incoming request, e.g.:
>>>> >>
>>>> >>   keystoneclient.middleware.auth_token [INFO]: Using Keystone
>>>> memcache for
>>>> >> caching token
>>>> >>
>>>> >> You may also want to use auth_token from keystoneclient >= v0.2.0 if
>>>> >> you're not already (instead of from keystone itself).
>>>> >>
>>>> >>
>>>> >> -Dolph
>>>> >>
>>>> >>
>>>> >> On Mon, Jan 14, 2013 at 11:43 AM, Leander Bessa Beernaert
>>>> >> <leanderbb at gmail.com> wrote:
>>>> >>>
>>>> >>> Are you by any chance referring to this topic
>>>> >>> https://lists.launchpad.net/openstack/msg08639.html regarding the
>>>> keystone
>>>> >>> token cache? If so I've already added the configuration line and
>>>> have not
>>>> >>> noticed any speedup :/
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> On Mon, Jan 14, 2013 at 5:19 PM, Leander Bessa Beernaert
>>>> >>> <leanderbb at gmail.com> wrote:
>>>> >>>>
>>>> >>>> I'm using the ubuntu 12.04 packages of the folsom repository by
>>>> the way.
>>>> >>>>
>>>> >>>>
>>>> >>>> On Mon, Jan 14, 2013 at 5:18 PM, Chuck Thier <cthier at gmail.com>
>>>> wrote:
>>>> >>>>>
>>>> >>>>> On Mon, Jan 14, 2013 at 11:03 AM, Leander Bessa Beernaert
>>>> >>>>> <leanderbb at gmail.com> wrote:
>>>> >>>>> > Also, I'm unable to run the swift-bench with keystone.
>>>> >>>>> >
>>>> >>>>>
>>>> >>>>> Hrm... That was supposed to be fixed with this bug:
>>>> >>>>> https://bugs.launchpad.net/swift/+bug/1011727
>>>> >>>>>
>>>> >>>>> My keystone dev instance isn't working at the moment, but I'll
>>>> see if
>>>> >>>>> I can get one of the team to take a look at it.
>>>> >>>>>
>>>> >>>>> --
>>>> >>>>> Chuck
>>>> >>>>
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>> _______________________________________________
>>>> >>> Mailing list: https://launchpad.net/~openstack
>>>> >>> Post to     : openstack at lists.launchpad.net
>>>> >>> Unsubscribe : https://launchpad.net/~openstack
>>>> >>> More help   : https://help.launchpad.net/ListHelp
>>>> >>>
>>>> >>
>>>> >
>>>>
>>>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~openstack
>>> Post to     : openstack at lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~openstack
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130115/a0d0e619/attachment.html>


More information about the Openstack mailing list