[Openstack-operators] Keystone upgrade issues

Jonathan Proulx jon at csail.mit.edu
Thu Aug 25 15:34:18 UTC 2016


Hi Matt,

Thanks for the pointers

On Thu, Aug 25, 2016 at 11:05:04AM -0400, Matt Fischer wrote:
:Jonathan,
:
:Are you using caching for tokens (not the middleware cache but keystone
:cache)? There's a bug in the caching so that when it tries to read the
:cache and unpack the token its missing some fields. It's been fixed and
:backported but may not be in your packages:
:https://bugs.launchpad.net/keystone/+bug/1592169

I am using memcache but this is with fresh tokens (and my test system
has been running longer than my token life at this point.

Interesting new twist.  If I get a toake with (openstack token issue)
then use curl myself:

curl -g -i -X GET https://keystone:35358/v2.0/users -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: $TOKEN"

I DO get a list of users. This is running locally on the keystone
controller node.  My bets are still on 'user error' not an actuall bug
for this one.

:Until that is fixed you can just flush memcache in a loop during the
:upgrade.

This is good to know for the production uprade (as are te following
bugs)

Thanks,
-Jon

:Also - heads-up that you will have this issue if you use caching in Mitaka
:that will lead to intermittent API call failures -
:https://bugs.launchpad.net/keystone/+bug/1600394
:
:And finally, this Cinder bug will show up once you're on Keystone Mitaka:
:https://bugs.launchpad.net/cinder/+bug/1597045
:
:
:
:On Thu, Aug 25, 2016 at 10:55 AM, Jonathan Proulx <jon at csail.mit.edu> wrote:
:
:> Hi All,
:>
:> working on testing our Kilo-> Mitaka keystone upgrade, and I've
:> clearly missied something I need to do or undo.
:>
:> After DB migration and the edits I belive are required to paste and
:> conf files I can get tokens (using password auth) but it won't seem to
:> accept them (for example with an admin user I get 'action requires
:> authorization' errors when trying to show users )
:>
:> Current setup is pretty simple and past upgrades of keystone have been
:> super easy, so other that reread and recheck not sure where I should
:> focus my attention.
:>
:> using:
:> fernet tokens
:> mysql local users
:> apache/wsgi
:> Ubuntu 14.04 cloud archive packages
:>
:> This is what I can see with --debug the client (both
:> python-keystoneclient and python-openstackclient) after getting the
:> initial auth token through password exchange:
:>
:> REQ: curl -g -i -X GET https://controller:35358/v2.0/users -H
:> "User-Agent: python-keystoneclient" -H "Accept: application/json" -H
:> "X-Auth-Token: {SHA1}<redacted>"
:> "GET /v2.0/users HTTP/1.1" 401 114
:> RESP: [401] Content-Length: 114 Vary: X-Auth-Token Keep-Alive: timeout=5
:> Server: Apache/2.4.7 (Ubuntu) Connection: Keep-Alive Date: Thu, 25 Aug 2016
:> 14:41:26 GMT WWW-Authenticate: Keystone uri="https://nimbus.csail.mit.
:> edu:35358" Content-Type: application/json X-Distribution: Ubuntu
:> RESP BODY: {"error": {"message": "The request you have made requires
:> authentication.", "code": 401, "title": "Unauthorized"}}
:>
:> (v3 requests are similar modulo API differences)
:>
:> Keysote.log in debug mode issues a couple deprecation warnings but no
:> errors (http://pastebin.com/WriB6u6i).  Not this log is for the same
:> event but response is UTC where log is local time (-0400)
:>
:> Any pointer to where I should focus my investigations woudl be most
:> welcome :)
:>
:> Thanks,
:> -Jon
:>
:> _______________________________________________
:> OpenStack-operators mailing list
:> OpenStack-operators at lists.openstack.org
:> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
:>

-- 



More information about the OpenStack-operators mailing list