[Openstack] [Swift] Intermittent error 403 "Access was denied to this resource"

Andrii Loshkovskyi loshkovskyi at gmail.com
Tue Jun 4 12:41:48 UTC 2013 

Thank you for answer.

Chmouel, do you mean the auth_token on Keystone or swift proxy server?

from /etc/keystone/keystone.conf

[filter:token_auth]
paste.filter_factory = keystone.middleware:TokenAuthMiddleware.factory

from /etc/swift/proxy-server.conf

[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
...
memcache_servers = 127.0.0.1:11211

Further debugging proved that hosts without memcached don't return the
error 403. I'm still investigating what service can return such error body
message.



On Tue, Jun 4, 2013 at 12:55 PM, Chmouel Boudjnah <chmouel at enovance.com>wrote:

> I have seen this when keystone is too busy for validating tokens.
> getting keystone behind apache or scaling up keystone make things a
> better (and make sure you are using swift memcache connection in
> auth_token).
>
>
> Chmouel.
>
> On Mon, Jun 3, 2013 at 8:15 PM, Andrii Loshkovskyi
> <loshkovskyi at gmail.com> wrote:
> > Hello,
> >
> > I would appreciate if you help me to troubleshoot the following issue:
> >
> > I am having error 403 intermittenly when listing containers in swift.
> > Sometimes the error appears a few times per hour, sometimes once per day.
> > Basically, it's possible to reproduce the error with a simple curl
> command:
> >
> > curl --get -v -H 'X-Auth-Token: ef644...'
> > http://swift-proxy.example.com:8080/v1/AUTH_323d0...
> > <body>
> > <h1>403 Forbidden</h1>
> > Access was denied to this resource.<br /><br />
> > </body>
> >
> > The token and swift proxy endpoint are all correct as most of the time
> the
> > command works.
> >
> > A few words about infrastructure: I use swift 1.7.4 and several swift
> > proxies. Users are authenticated via Keystone. Tokens are cached with
> > memcached on swift proxy servers.
> >
> > I did a lot of tests to figure out what service generates such error:
> >
> > - same issue happens with each swift proxy server, with or without
> memcached
> > enabled
> > - it happens with different users and in different tenants
> > - I downloaded sources of swift and Keystone and grepped on that error.
> > There are some HTTPForbidden values returned in code but no one with the
> > body 'Access denied to this resource'
> > - I tried monitoring traffic with tcpdump to catch the error and
> understand
> > who's sending it but with no success yet
> > - the issue might be related to swift ACL rules but I haven't set any
> > read/write permissions for containers
> > - set debug logs for swift proxy but nothing has been found yet
> >
> > Please help me to understand how this error is returned. Thank you for
> your
> > time.
> >
> >
> > --
> > Kind regards,
> > Andrii Loshkovskyi
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to     : openstack at lists.launchpad.net
> > Unsubscribe : https://launchpad.net/~openstack
> > More help   : https://help.launchpad.net/ListHelp
> >
>



-- 
Kind regards,
Andrii Loshkovskyi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130604/4c6f2531/attachment.html>

More information about the Openstack mailing list