Open Stack

Comments in-line.

-----Original Message-----
From: Joshua Harlow <harlowja at outlook.com>
Reply: OpenStack Development Mailing List (not for usage questions) <openstack-dev at lists.openstack.org>>
Date: September 29, 2014 at 21:52:20
To: OpenStack Development Mailing List (not for usage questions) <openstack-dev at lists.openstack.org>>
Subject:  Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift?

> +1 Lets not continue to expand the usage of persisted tokens :-/

We can look at moving away from persisted tokens, but as of the Kilo cycle there is no plan to eliminate UUID tokens (or PKI/PKIZ tokens for that matter). This means that we will need to continue to support them. If there is a legitimate desire to utilize swift or something similar and make this experience better, it behooves us to consider accepting these improvements. I believe that non-persistent tokens are a great place to strive for, but it won’t meet every deployment need. Moreover, currently PKI tokens (the “persistent-less” capable version) adds significant overhead to each request. While 5k-10k of data doesn’t seem like a lot, when there is a large volume of actions taken it adds up fast. This is putting the onus on the end user to acquire the “authorization” and transmit that same information over and over again to each of the APIs (especially in the swift use-case of many, many, many small object requests). 5-10k for a request consisting of a few hundred bytes is a significant overhead. Asking a service local to your datacenter might be in some cases more efficient and produce less edge traffic.

There is a reason why websites often use separate domains (top-level) for their CDNs rather than just sourcing from the primary domain. This is to (especially in the case of large cookies such as are generated by Social Media sites) avoid having to transit the primary user cookie when gathering an asset, especially if that asset is publicly viewable. 

> We should be trying to move away from such types of persistence and its associated complexity  
> IMHO. Most major websites don't need to have tokens that are saved around regions in there  
> internal backend databases, just so people can use a REST webservice (or a website...)  
> so I would hope that we don't need to (if it works for major websites why doesn't it work  
> for us?).

As of today, PKI tokens (and even UUID token bodies) convey a lot of information. In fact, the tokens convey all the information (except if the token is revoked) needed to interact with the API endpoint. A lot of the “big” web apps will not encode all of that information into the signed cookie, some of that information can be sourced directly (often things like ACL-like-constructs or RBAC information won’t be in the cookie itself). PKI tokens are also significantly more difficult to deploy (needing certificates, CAs, and ensuring everything is in sync is definitely more work than saying “talk to URL xxx and ask if the token is valid”; you’re trading one set of operational concerns and scaling points for another).

Now if we start looking at OAuth and some of the other SSO technology we start getting a little bit closer to what we need to be able to eliminate the backing store with the exception of two critical items:

1) OAuth-type workflow would be a reauthorization for each endpoint (provided they are on separate domains/hosts/etc, which is the case for each of our services). You don’t (for example) Authenticate with UbuntuOne for https://review.openstack.org and without re-authenticating go to the wiki pages and edit the wiki. The wiki, while it uses the same authentication source, same identity source, and is even on an associated host/domain, requires a re-auth to use.

If you were to get an SSO login to Keystone and then each endpoint requires another bounce back to Keystone for AuthZ it isn’t really making the REST API any more usable nor making the user experience any better.

2) All of the OpenStack specific permissions are not necessarily conveyed by the Identity Provider (Keystone is not typically the Identity provider in this case), and most Identity Providers do not have the concept of Projects, Domains, or even Roles that are always going to make sense within OpenStack. We’re still back to needing to ask for information from Keystone.


> My 2 cents is that we should really think about why this is needed and why we can't operate  
> using signed-cookie like mechanisms (after all it works for everyone else). If cross-region  
> tokens are a problem, then maybe we should solve the root of the issue (having a token that  
> works across regions) so that no replication is needed at all…

With PKI tokens (if you share the signing certificate) it is possible to handle cross region support. There are some limitations here (such as possibly needing to replicate the Identity/Assignment information between regions so an Auth in region X would work in region Y as well as vice-versa).

There are other options to help address these types of issues but I think we need to sit down and decide what our real end goal is. Do we want PKI tokens? Do we want UUID tokens? Do we want something totally different? I don’t have the answers here, it’s a topic with a lot of conversation and not a good answer as of today.


To summarize, tokens are not going away (for a number of reasons outlined both here and in other conversations). It all comes down to deployment specific usage patterns. The important questions are:

* What is an acceptable token size for a given deployment? How do we get to a reasonable point for *most* deployments (sane defaults)?
* What is an acceptable frequency of requests against Keystone for validation? Including the token revocation list, revocation events, certs, token validation, etc.
* How do we ensure the options for deployment meet the needs of our users? Does a swift persistence backend help meet the deployer’s needs? If so is it worth having in-tree?


> On Sep 29, 2014, at 8:22 PM, Adam Young wrote:
>  
> > On 09/29/2014 12:12 PM, Jay Pipes wrote:
> >> Hey Stackers,
> >>
> >> So, I had a thought this morning (uh-oh, I know...).
> >>
> >> What if we wrote a token driver in Keystone that uses Swift for backend storage?
> >>
> >> I have long been an advocate of the memcache token driver versus the SQL driver for performance  
> reasons. However, the problem with the memcache token driver is that if you want to run  
> multiple OpenStack regions, you could share the identity data in Keystone using replicated  
> database technology (mysql galera/PXC, pgpool II, or even standard mysql master/slave),  
> but each region needs to have its own memcache service for tokens. This means that tokens  
> are not shared across regions, which means that users have to log in separately to each  
> region's dashboard.
> >>
> >> I personally considered this a tradeoff worth accepting. But then, today, I thought...  
> what about storing tokens in a globally-distributed Swift cluster? That would take  
> care of the replication needs automatically, since Swift would do the needful. And,  
> add to that, Swift was designed for storing lots of small objects, which tokens are...  
> >>
> >> Thoughts? I think it would be a cool dogfooding effort if nothing else, and give users  
> yet another choice in how they handle multi-region tokens.
> >
> > Um...I hate all persisted tokens. This takes them to a new level of badness.
> >
> > Do we really need this?
> >
> >