Open Stack

On Wed, 10 Sep 2014 10:14:32 AM Sean Dague wrote:
> Going through the untriaged Nova bugs, and there are a few on a similar
> pattern:
> 
> Nova operation in progress.... takes a while
> Crosses keystone token expiration time
> Timeout thrown
> Operation fails
> Terrible 500 error sent back to user
> 
> It seems like we should have a standard pattern that on token expiration
> the underlying code at least gives one retry to try to establish a new
> token to complete the flow, however as far as I can tell *no* clients do
> this.

Just because this came up in conversation a few weeks ago in the context of 
the ironic client.  I've read some docs and written a keystone client, but I'm 
not super-familiar with keystone internals - apologies if I miss something 
fundamental.


There are two broadly different approaches to dealing with this:

(As described by Sean, and implemented in a few clients)
1. At the bottom layer, try to refresh the token and immediately retry 
whenever a server response indicates the token has expired.

(As inspired by eg kerberos)
2. Ensure at some environmental/top layer that the advertised token lifetime 
exceeds the timeout set on the request, before making the request.  This 
implies (since there's no special handling in place) failing if the token was 
expired earlier than expected.


The primary distinction being that in (2) the client is ignorant of how to 
create tokens, and just assumes they're valid.

(2) is particularly easy to code for simple "one shot" command line clients.  
For a persistent client, the easiest approach is probably to have an 
asynchronous loop that just keeps refreshing the stored token whenever it 
approaches "expiry - max_single_request_timeout".

My concern with (1) is that it involves passing username/password all the way 
down to the bottom layers - see the heat example where this means crossing 
into another program/service.  Moreover, if the token was expired earlier than 
advertised it probably means the admin has deliberately rejected the user or 
something and the intent is that they _should_ be locked out - it would be 
unfortunate to have a synchronised retry attack on keystone from all the 
rejected clients at that point :/

-- 
 - Gus